[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:09 dist-test-slave-twwt krb5kdc[27995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:09 dist-test-slave-twwt krb5kdc[27995](info): set up 2 sockets
Aug 08 15:48:09 dist-test-slave-twwt krb5kdc[27995](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260808 15:48:11.107136 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:11 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:11.316833 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43445
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:11.666025 28015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:11.666335 28015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:11.666497 28015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:11.673806 28015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:11.673944 28015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:11.674057 28015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:11.674154 28015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:11.674250 28015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:11.684499 28015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:11.686035 28015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:11.687536 28015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:11.700265 28021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:11.700332 28022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:11.700332 28024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:11.700773 28015 server_base.cc:1034] running on GCE node
I20260808 15:48:11.701951 28015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:11.703629 28015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:11.704916 28015 hybrid_clock.cc:648] HybridClock initialized: now 1786204091704867 us; error 112 us; skew 500 ppm
I20260808 15:48:11.705339 28015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:11 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204091, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:11.709154 28015 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:11.710923 28015 webserver.cc:460] Webserver started at http://127.27.84.254:45843/ using document root <none> and password file <none>
I20260808 15:48:11.711686 28015 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:11.711838 28015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:11.712180 28015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:11.715521 28015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "02fd35d02e384ddb95c6913a7a76cfcc"
format_stamp: "Formatted at 2026-08-08 15:48:11 on dist-test-slave-twwt"
server_key: "904f0d6cf7d5a0e031f019cb6f510a73"
server_key_iv: "a3dfdb4c1cccf2e27f5879bf955e7c32"
server_key_version: "encryptionkey@0"
I20260808 15:48:11.716361 28015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "02fd35d02e384ddb95c6913a7a76cfcc"
format_stamp: "Formatted at 2026-08-08 15:48:11 on dist-test-slave-twwt"
server_key: "904f0d6cf7d5a0e031f019cb6f510a73"
server_key_iv: "a3dfdb4c1cccf2e27f5879bf955e7c32"
server_key_version: "encryptionkey@0"
I20260808 15:48:11.721809 28015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:48:11.725556 28031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:11.727058 28015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:11.727252 28015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "02fd35d02e384ddb95c6913a7a76cfcc"
format_stamp: "Formatted at 2026-08-08 15:48:11 on dist-test-slave-twwt"
server_key: "904f0d6cf7d5a0e031f019cb6f510a73"
server_key_iv: "a3dfdb4c1cccf2e27f5879bf955e7c32"
server_key_version: "encryptionkey@0"
I20260808 15:48:11.727468 28015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:11.753597 28015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:11.759388 28015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:11.759761 28015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:11.781818 28015 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43445
I20260808 15:48:11.781822 28118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43445 every 8 connection(s)
I20260808 15:48:11.783476 28015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:11.787669 28119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:11.788503 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28015
I20260808 15:48:11.788866 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:11.791504 27987 external_mini_cluster.cc:1428] Setting key ba652746ddff8aca1bda33e1457b2059
I20260808 15:48:11.801117 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc: Bootstrap starting.
I20260808 15:48:11.804456 28119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:11.805649 28119 log.cc:824] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:11.808785 28119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc: No bootstrap required, opened a new log
I20260808 15:48:11.816289 28119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43445 } }
I20260808 15:48:11.816710 28119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:11.816828 28119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fd35d02e384ddb95c6913a7a76cfcc, State: Initialized, Role: FOLLOWER
I20260808 15:48:11.817484 28119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43445 } }
I20260808 15:48:11.817734 28119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:11.817891 28119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:11.818089 28119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:11.821727 28119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43445 } }
Aug 08 15:48:11 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:11.822326 28119 leader_election.cc:302] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02fd35d02e384ddb95c6913a7a76cfcc; no voters:
I20260808 15:48:11.823053 28119 leader_election.cc:288] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:11.823171 28126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:11.824414 28126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [term 1 LEADER]: Becoming Leader. State: Replica: 02fd35d02e384ddb95c6913a7a76cfcc, State: Running, Role: LEADER
I20260808 15:48:11.825493 28126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43445 } }
I20260808 15:48:11.825709 28119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:11.828970 28128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fd35d02e384ddb95c6913a7a76cfcc. Latest consensus state: current_term: 1 leader_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43445 } } }
I20260808 15:48:11.829319 28128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:11.829852 28127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fd35d02e384ddb95c6913a7a76cfcc" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43445 } } }
I20260808 15:48:11.830183 28127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:11.830438 28137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:11.837621 28122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:11.796480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.27.84.254:43445)
0808 15:48:11.797469 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:11.797486 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:11.798973 (+ 1487us) server_negotiation.cc:407] Connection header received
0808 15:48:11.808369 (+ 9396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:11.808392 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:11.808764 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:11.809247 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:11.812296 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:11.813708 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:11.815476 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:11.816155 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:11.816370 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:11.823985 (+ 7615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:11.823996 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:11.824017 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:11.824086 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:11.828687 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:11.830133 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:11.830147 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:11.830160 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:11.830480 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:11.831513 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:11.831521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:11.831527 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:11.832761 (+ 1234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:11.833214 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:11.834999 (+ 1785us) server_negotiation.cc:299] Negotiation successful
0808 15:48:11.835643 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":428,"threads_started":1}
I20260808 15:48:11.841274 28137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:11.852450 28137 catalog_manager.cc:1347] Generated new cluster ID: 741d4f2997a849c0a2f553928b380ab8
I20260808 15:48:11.852610 28137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:11.871771 28137 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:11.873196 28137 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:11.885401 28137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02fd35d02e384ddb95c6913a7a76cfcc: Generated new TSK 0
I20260808 15:48:11.886334 28137 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:11.951121 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43445
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:12.338503 28156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:12.338969 28156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:12.339135 28156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:12.347196 28156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:12.347359 28156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:12.347587 28156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:12.357800 28156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43445
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:12.359570 28156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:12.361032 28156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:12.374717 28164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:12.374629 28165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:12.374661 28168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:12.375705 28156 server_base.cc:1034] running on GCE node
I20260808 15:48:12.376308 28156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:12.377257 28156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:12.378458 28156 hybrid_clock.cc:648] HybridClock initialized: now 1786204092378414 us; error 30 us; skew 500 ppm
I20260808 15:48:12.378824 28156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:12 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204092, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:12.382185 28156 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:12.383690 28156 webserver.cc:460] Webserver started at http://127.27.84.193:34183/ using document root <none> and password file <none>
I20260808 15:48:12.384383 28156 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:12.384490 28156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:12.384755 28156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:12.387905 28156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "aae78b260c604fec9c5fb26d52ee2b0e"
format_stamp: "Formatted at 2026-08-08 15:48:12 on dist-test-slave-twwt"
server_key: "6325977d897a4fc4253fc47585a86ff0"
server_key_iv: "21e90dab6329ae58bb5fbe6630a4006a"
server_key_version: "encryptionkey@0"
I20260808 15:48:12.388674 28156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "aae78b260c604fec9c5fb26d52ee2b0e"
format_stamp: "Formatted at 2026-08-08 15:48:12 on dist-test-slave-twwt"
server_key: "6325977d897a4fc4253fc47585a86ff0"
server_key_iv: "21e90dab6329ae58bb5fbe6630a4006a"
server_key_version: "encryptionkey@0"
I20260808 15:48:12.393774 28156 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:48:12.397228 28179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:12.398481 28156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:12.398661 28156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "aae78b260c604fec9c5fb26d52ee2b0e"
format_stamp: "Formatted at 2026-08-08 15:48:12 on dist-test-slave-twwt"
server_key: "6325977d897a4fc4253fc47585a86ff0"
server_key_iv: "21e90dab6329ae58bb5fbe6630a4006a"
server_key_version: "encryptionkey@0"
I20260808 15:48:12.398857 28156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:12.427031 28156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:12.431770 28156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:12.432099 28156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:12.433162 28156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:12.434562 28156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:12.434653 28156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 15:48:12.434826 28156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:12.434906 28156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:12.469811 28156 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43775
I20260808 15:48:12.469862 28348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43775 every 8 connection(s)
I20260808 15:48:12.471256 28156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:12.471801 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28156
I20260808 15:48:12.472061 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:12.472954 27987 external_mini_cluster.cc:1428] Setting key 490fbd57a35065ee0f15ee5faf8245da
Aug 08 15:48:12 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204092, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
I20260808 15:48:12.496213 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:12.474409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55235 (local address 127.27.84.254:43445)
0808 15:48:12.475007 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:12.475014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:12.476293 (+ 1279us) server_negotiation.cc:407] Connection header received
0808 15:48:12.479468 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:12.479473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:12.479541 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:12.479628 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:12.481820 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:12.482610 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:12.484234 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:12.484986 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:12.485137 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:12.488941 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:12.488950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:12.488957 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:12.489004 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:12.492906 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:12.493683 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:12.493690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:12.493695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:12.493868 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:12.494291 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:12.494297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:12.494301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:12.494733 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:12.494909 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:12.495588 (+ 679us) server_negotiation.cc:299] Negotiation successful
0808 15:48:12.495824 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":388,"threads_started":1}
I20260808 15:48:12.497341 28352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:12.475224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43445 (local address 127.27.84.193:55235)
0808 15:48:12.476114 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:12.476167 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:12.479144 (+ 2977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:12.479925 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:12.479940 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:12.480761 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:12.481635 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:12.481665 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:12.482849 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:12.482853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:12.484098 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:12.484115 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:12.485213 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:12.485217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:12.485444 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:12.486176 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:12.486200 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:12.488687 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:12.493076 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:12.493084 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:12.493098 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:12.493553 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:12.493983 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:12.493986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:12.493989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:12.494173 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:12.494864 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:12.494871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:12.495196 (+ 325us) client_negotiation.cc:769] Sending connection context
0808 15:48:12.495480 (+ 284us) client_negotiation.cc:240] Negotiation successful
0808 15:48:12.496065 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
I20260808 15:48:12.499229 28349 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43445
I20260808 15:48:12.499620 28349 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:12.500414 28349 heartbeater.cc:511] Master 127.27.84.254:43445 requested a full tablet report, sending...
I20260808 15:48:12.502987 28059 ts_manager.cc:194] Registered new tserver with Master: aae78b260c604fec9c5fb26d52ee2b0e (127.27.84.193:43775)
I20260808 15:48:12.505095 28059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:55235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:12.525586 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43445
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:12.890416 28359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:12.890790 28359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:12.890914 28359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:12.898589 28359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:12.898761 28359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:12.898898 28359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:12.909109 28359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43445
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:12.910385 28359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:12.911809 28359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:12.925836 28367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:12.925809 28370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:12.925871 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:12.926508 28359 server_base.cc:1034] running on GCE node
I20260808 15:48:12.927311 28359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:12.928252 28359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:12.929462 28359 hybrid_clock.cc:648] HybridClock initialized: now 1786204092929420 us; error 26 us; skew 500 ppm
I20260808 15:48:12.929802 28359 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:12 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204092, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:12.933321 28359 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:12.934846 28359 webserver.cc:460] Webserver started at http://127.27.84.194:39343/ using document root <none> and password file <none>
I20260808 15:48:12.935491 28359 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:12.935597 28359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:12.935874 28359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:12.938807 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "5c0ff5246e854392975c44c6f05820be"
format_stamp: "Formatted at 2026-08-08 15:48:12 on dist-test-slave-twwt"
server_key: "f0f4885127f37da0fc0cb02c189d1693"
server_key_iv: "77c5d007904ad8d8cf37fd11e384e791"
server_key_version: "encryptionkey@0"
I20260808 15:48:12.939544 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "5c0ff5246e854392975c44c6f05820be"
format_stamp: "Formatted at 2026-08-08 15:48:12 on dist-test-slave-twwt"
server_key: "f0f4885127f37da0fc0cb02c189d1693"
server_key_iv: "77c5d007904ad8d8cf37fd11e384e791"
server_key_version: "encryptionkey@0"
I20260808 15:48:12.944558 28359 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:48:12.948139 28387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:12.949450 28359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:12.949584 28359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "5c0ff5246e854392975c44c6f05820be"
format_stamp: "Formatted at 2026-08-08 15:48:12 on dist-test-slave-twwt"
server_key: "f0f4885127f37da0fc0cb02c189d1693"
server_key_iv: "77c5d007904ad8d8cf37fd11e384e791"
server_key_version: "encryptionkey@0"
I20260808 15:48:12.949757 28359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:12.978976 28359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:12.986249 28359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:12.986572 28359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:12.987659 28359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:12.989066 28359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:12.989146 28359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:12.989293 28359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:12.989377 28359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:13.025532 28359 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:45185
I20260808 15:48:13.025585 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:45185 every 8 connection(s)
I20260808 15:48:13.027240 28359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:13.036291 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28359
I20260808 15:48:13.036520 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:13.037155 27987 external_mini_cluster.cc:1428] Setting key dadea27b0dd9578ad6269a0632b73cb9
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204092, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:13.050582 28567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.030035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:45847 (local address 127.27.84.254:43445)
0808 15:48:13.030678 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.030685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.031679 (+ 994us) server_negotiation.cc:407] Connection header received
0808 15:48:13.033736 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.033751 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.033852 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.033975 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.035756 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.036560 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.038897 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.039579 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.039680 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.044217 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.044222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.044226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.044259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.047123 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.047993 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.048004 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.048007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.048151 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.048708 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.048720 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.048722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.049010 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.049172 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.049944 (+ 772us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.050191 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":367,"threads_started":1}
I20260808 15:48:13.051714 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.030545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43445 (local address 127.27.84.194:45847)
0808 15:48:13.031467 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:13.031519 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:13.033431 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:13.034253 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:13.034263 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:13.034918 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:13.035550 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:13.035570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.036788 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.036794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:13.038748 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:13.038762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.039830 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.039839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:13.040152 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.040969 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:13.041015 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:13.043948 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:13.047328 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:13.047338 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:13.047377 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:13.047844 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:13.048320 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:13.048323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:13.048326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:13.048582 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:13.049146 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:13.049155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:13.049549 (+ 394us) client_negotiation.cc:769] Sending connection context
0808 15:48:13.049832 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 15:48:13.050441 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":306,"threads_started":1}
I20260808 15:48:13.053335 28565 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43445
I20260808 15:48:13.053702 28565 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:13.054759 28565 heartbeater.cc:511] Master 127.27.84.254:43445 requested a full tablet report, sending...
I20260808 15:48:13.056871 28059 ts_manager.cc:194] Registered new tserver with Master: 5c0ff5246e854392975c44c6f05820be (127.27.84.194:45185)
I20260808 15:48:13.058125 28059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:45847
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:13.087950 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43445
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:13.436408 28572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:13.436784 28572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:13.436954 28572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:13.444438 28572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:13.444597 28572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:13.444795 28572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:13.455209 28572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43445
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:13.456580 28572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:13.458098 28572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:13.472249 28585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:13.472280 28582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:13.472903 28583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:13.478803 28572 server_base.cc:1034] running on GCE node
I20260808 15:48:13.479458 28572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:13.480513 28572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:13.481748 28572 hybrid_clock.cc:648] HybridClock initialized: now 1786204093481705 us; error 54 us; skew 500 ppm
I20260808 15:48:13.482084 28572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:13.485369 28572 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:13.487027 28572 webserver.cc:460] Webserver started at http://127.27.84.195:41303/ using document root <none> and password file <none>
I20260808 15:48:13.487792 28572 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:13.487948 28572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:13.488279 28572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:13.491336 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "072dedf9dc5e48fab9d94e72fd9c6a14"
format_stamp: "Formatted at 2026-08-08 15:48:13 on dist-test-slave-twwt"
server_key: "fc624e537b49365347ad131ee43ad684"
server_key_iv: "9a066b00fbe983185ae7fca849d052eb"
server_key_version: "encryptionkey@0"
I20260808 15:48:13.492133 28572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "072dedf9dc5e48fab9d94e72fd9c6a14"
format_stamp: "Formatted at 2026-08-08 15:48:13 on dist-test-slave-twwt"
server_key: "fc624e537b49365347ad131ee43ad684"
server_key_iv: "9a066b00fbe983185ae7fca849d052eb"
server_key_version: "encryptionkey@0"
I20260808 15:48:13.497525 28572 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:48:13.501080 28595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:13.502668 28572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:48:13.502858 28572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "072dedf9dc5e48fab9d94e72fd9c6a14"
format_stamp: "Formatted at 2026-08-08 15:48:13 on dist-test-slave-twwt"
server_key: "fc624e537b49365347ad131ee43ad684"
server_key_iv: "9a066b00fbe983185ae7fca849d052eb"
server_key_version: "encryptionkey@0"
I20260808 15:48:13.503065 28572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:13.508278 28349 heartbeater.cc:503] Master 127.27.84.254:43445 was elected leader, sending a full tablet report...
I20260808 15:48:13.519598 28572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:13.524115 28572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:13.524432 28572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:13.525452 28572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:13.527009 28572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:13.527084 28572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:13.527232 28572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:13.527331 28572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:13.563261 28572 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:34235
I20260808 15:48:13.563316 28753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:34235 every 8 connection(s)
I20260808 15:48:13.564643 28572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:13.569185 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28572
I20260808 15:48:13.569389 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:13.569870 27987 external_mini_cluster.cc:1428] Setting key d648647951631c796d873934ce10fcae
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:13.585744 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.567834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:38399 (local address 127.27.84.254:43445)
0808 15:48:13.568383 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.568388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.569284 (+ 896us) server_negotiation.cc:407] Connection header received
0808 15:48:13.571736 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.571741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.571832 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.571960 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.573623 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.574440 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.576100 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.576490 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.576626 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.580099 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.580104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.580108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.580140 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.582556 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.583383 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.583392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.583395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.583514 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.583973 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.583978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.583980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.584240 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.584420 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.585107 (+ 687us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.585326 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":273,"threads_started":1}
I20260808 15:48:13.586908 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.568275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43445 (local address 127.27.84.195:38399)
0808 15:48:13.569095 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:13.569146 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:13.571362 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:13.572288 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:13.572299 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:13.572866 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:13.573444 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:13.573463 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.574651 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.574660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:13.575934 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:13.575947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.576720 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.576727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:13.576975 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.577611 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:13.577633 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:13.579879 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:13.582754 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:13.582763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:13.582780 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:13.583243 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:13.583630 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:13.583635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:13.583639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:13.583833 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:13.584377 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:13.584387 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:13.584729 (+ 342us) client_negotiation.cc:769] Sending connection context
0808 15:48:13.585004 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 15:48:13.585538 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":310,"threads_started":1}
I20260808 15:48:13.588215 28754 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43445
I20260808 15:48:13.588534 28754 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:13.589289 28754 heartbeater.cc:511] Master 127.27.84.254:43445 requested a full tablet report, sending...
I20260808 15:48:13.590821 28059 ts_manager.cc:194] Registered new tserver with Master: 072dedf9dc5e48fab9d94e72fd9c6a14 (127.27.84.195:34235)
I20260808 15:48:13.591662 28059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:38399
I20260808 15:48:13.599234 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:13.651708 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.635836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.27.84.193:43775)
0808 15:48:13.636653 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.636661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.636686 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:48:13.636748 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.636754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.636983 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.637166 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.638749 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.639811 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.641115 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.641703 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.641845 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.645831 (+ 3986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.645843 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.645858 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.645898 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.649007 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.649671 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.649678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.649680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.649813 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.650148 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.650151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.650153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.650489 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.650774 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.651075 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.651309 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":381,"threads_started":1}
W20260808 15:48:13.653062 28226 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45392
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:13.703404 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.689202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.27.84.193:43775)
0808 15:48:13.689475 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.689480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.689621 (+ 141us) server_negotiation.cc:407] Connection header received
0808 15:48:13.689778 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.689782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.689883 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.689973 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.691431 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.692151 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.693377 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.693839 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.693944 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.697923 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.697932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.697938 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.697987 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.700865 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.701596 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.701599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.701602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.701730 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.702089 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.702093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.702095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.702348 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.702695 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.702899 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.703076 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 15:48:13.705101 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28156
I20260808 15:48:13.718402 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.706084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.27.84.193:43775)
0808 15:48:13.706293 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.706300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.706668 (+ 368us) server_negotiation.cc:407] Connection header received
0808 15:48:13.706846 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.706851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.706905 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.707007 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.708191 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.708875 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.710181 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.710597 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.710792 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.713023 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.713028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.713031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.713060 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.715770 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.716438 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.716444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.716448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.716602 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.716991 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.716996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.717006 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.717262 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.717514 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.717818 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.718040 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:48:13.719220 28344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:13.809576 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28156
I20260808 15:48:13.837639 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28359
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:13.853796 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.838717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.27.84.194:45185)
0808 15:48:13.839319 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.839326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.839354 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:48:13.839421 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.839428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.839652 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.839791 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.840991 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.842043 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.843398 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.843954 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.844054 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.848196 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.848207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.848221 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.848260 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.851261 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.851837 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.851844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.851846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.851995 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.852342 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.852345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.852347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.852668 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.852927 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.853197 (+ 270us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.853439 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":321,"threads_started":1}
I20260808 15:48:13.855259 28556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:13.942152 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28359
I20260808 15:48:13.967955 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28572
Aug 08 15:48:13 dist-test-slave-twwt krb5kdc[27995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:13.985277 28779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:13.969247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.27.84.195:34235)
0808 15:48:13.969931 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:13.969938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:13.969968 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:48:13.970031 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:13.970037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:13.970256 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:13.970448 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:13.971850 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.972915 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.974250 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:13.974937 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:13.975085 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:13.979409 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:13.979422 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:13.979437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:13.979473 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:13.982255 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.983087 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.983094 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.983097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.983221 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:13.983621 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:13.983625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:13.983627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:13.983987 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:13.984312 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:13.984639 (+ 327us) server_negotiation.cc:299] Negotiation successful
0808 15:48:13.984926 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":364,"threads_started":1}
I20260808 15:48:13.986778 28748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:14.069873 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28572
I20260808 15:48:14.095595 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28015
I20260808 15:48:14.096822 28113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:14.203588 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28015
2026-08-08T15:48:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4475 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:14 dist-test-slave-twwt krb5kdc[28785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:14 dist-test-slave-twwt krb5kdc[28785](info): set up 2 sockets
Aug 08 15:48:14 dist-test-slave-twwt krb5kdc[28785](info): commencing operation
krb5kdc: starting...
W20260808 15:48:15.567019 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:15 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:15.787093 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36927
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:16.132591 28806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:16.132984 28806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:16.133162 28806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:16.141352 28806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:16.141548 28806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:16.141686 28806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:16.141819 28806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:16.141929 28806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:16.152768 28806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:16.154230 28806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:16.155673 28806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:16.167932 28816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:16.168145 28815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:16.167718 28819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:16.169167 28806 server_base.cc:1034] running on GCE node
I20260808 15:48:16.170135 28806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:16.171748 28806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:16.173026 28806 hybrid_clock.cc:648] HybridClock initialized: now 1786204096172975 us; error 43 us; skew 500 ppm
I20260808 15:48:16.173442 28806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:16 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204096, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:16.182206 28806 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:16.184132 28806 webserver.cc:460] Webserver started at http://127.27.84.254:34351/ using document root <none> and password file <none>
I20260808 15:48:16.184923 28806 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:16.185091 28806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:16.185446 28806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:16.188831 28806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "17e4e620b0f8422eb42dc28245c49d6c"
format_stamp: "Formatted at 2026-08-08 15:48:16 on dist-test-slave-twwt"
server_key: "5fbac70e47bd9b3de82adc4e17b5d60e"
server_key_iv: "92fe2e3448a4954b97cf24199e86832c"
server_key_version: "encryptionkey@0"
I20260808 15:48:16.189759 28806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "17e4e620b0f8422eb42dc28245c49d6c"
format_stamp: "Formatted at 2026-08-08 15:48:16 on dist-test-slave-twwt"
server_key: "5fbac70e47bd9b3de82adc4e17b5d60e"
server_key_iv: "92fe2e3448a4954b97cf24199e86832c"
server_key_version: "encryptionkey@0"
I20260808 15:48:16.195689 28806 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:48:16.199553 28828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:16.201040 28806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:16.201241 28806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "17e4e620b0f8422eb42dc28245c49d6c"
format_stamp: "Formatted at 2026-08-08 15:48:16 on dist-test-slave-twwt"
server_key: "5fbac70e47bd9b3de82adc4e17b5d60e"
server_key_iv: "92fe2e3448a4954b97cf24199e86832c"
server_key_version: "encryptionkey@0"
I20260808 15:48:16.201478 28806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:16.233099 28806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:16.237692 28806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:16.238025 28806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:16.257465 28806 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:36927
I20260808 15:48:16.257484 28910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:36927 every 8 connection(s)
I20260808 15:48:16.259033 28806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:16.262919 28911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:16.268982 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28806
I20260808 15:48:16.269227 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:16.269904 27987 external_mini_cluster.cc:1428] Setting key 7590ed246d97b117c200f6643d9ffc24
I20260808 15:48:16.273888 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c: Bootstrap starting.
I20260808 15:48:16.277774 28911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:16.278904 28911 log.cc:824] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:16.281987 28911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c: No bootstrap required, opened a new log
Aug 08 15:48:16 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:16.290262 28911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e4e620b0f8422eb42dc28245c49d6c" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36927 } }
I20260808 15:48:16.290675 28911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:16.290786 28911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e4e620b0f8422eb42dc28245c49d6c, State: Initialized, Role: FOLLOWER
I20260808 15:48:16.291411 28911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e4e620b0f8422eb42dc28245c49d6c" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36927 } }
I20260808 15:48:16.291676 28911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:16.291823 28911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:16.292019 28911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:16.294147 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:16.272119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.27.84.254:36927)
0808 15:48:16.272936 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:16.272947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:16.272988 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:48:16.274826 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:16.274850 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:16.275221 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:16.275566 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:16.277433 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:16.278869 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:16.280402 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:16.281070 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:16.281201 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:16.285893 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:16.285901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:16.285915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:16.285954 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:16.289193 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:16.289926 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:16.289934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:16.289940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:16.290105 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:16.290484 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:16.290489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:16.290503 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:16.291244 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:16.291575 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:16.292251 (+ 676us) server_negotiation.cc:299] Negotiation successful
0808 15:48:16.292746 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":358,"threads_started":1}
I20260808 15:48:16.295415 28911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e4e620b0f8422eb42dc28245c49d6c" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36927 } }
I20260808 15:48:16.295964 28911 leader_election.cc:302] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e4e620b0f8422eb42dc28245c49d6c; no voters:
I20260808 15:48:16.296609 28911 leader_election.cc:288] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:16.296806 28919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:16.298174 28919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [term 1 LEADER]: Becoming Leader. State: Replica: 17e4e620b0f8422eb42dc28245c49d6c, State: Running, Role: LEADER
I20260808 15:48:16.298748 28919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e4e620b0f8422eb42dc28245c49d6c" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36927 } }
I20260808 15:48:16.298978 28911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:16.301793 28923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e4e620b0f8422eb42dc28245c49d6c. Latest consensus state: current_term: 1 leader_uuid: "17e4e620b0f8422eb42dc28245c49d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e4e620b0f8422eb42dc28245c49d6c" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36927 } } }
I20260808 15:48:16.302182 28923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:16.302584 28920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17e4e620b0f8422eb42dc28245c49d6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e4e620b0f8422eb42dc28245c49d6c" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36927 } } }
I20260808 15:48:16.302975 28920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:16.305789 28928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:16.311380 28928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:16.320477 28928 catalog_manager.cc:1347] Generated new cluster ID: 7363c0fddd7844738bada74da7112a92
I20260808 15:48:16.320605 28928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:16.340411 28928 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:16.341828 28928 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:16.352587 28928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17e4e620b0f8422eb42dc28245c49d6c: Generated new TSK 0
I20260808 15:48:16.353649 28928 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:16.417893 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36927
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:16.769183 28949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:16.769490 28949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:16.769666 28949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:16.777230 28949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:16.777347 28949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:16.777534 28949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:16.787770 28949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:16.789170 28949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:16.790663 28949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:16.805258 28964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:16.806010 28968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:16.806133 28965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:16.811585 28949 server_base.cc:1034] running on GCE node
I20260808 15:48:16.812280 28949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:16.813382 28949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:16.814680 28949 hybrid_clock.cc:648] HybridClock initialized: now 1786204096814579 us; error 44 us; skew 500 ppm
I20260808 15:48:16.815028 28949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:16 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204096, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:16.818367 28949 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:16.820067 28949 webserver.cc:460] Webserver started at http://127.27.84.193:40573/ using document root <none> and password file <none>
I20260808 15:48:16.820808 28949 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:16.820959 28949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:16.821295 28949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:16.824349 28949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "75785750308649209a96fc5f2e61eb8c"
format_stamp: "Formatted at 2026-08-08 15:48:16 on dist-test-slave-twwt"
server_key: "4606199410ce6f7036ed1f691e466905"
server_key_iv: "110763c9baeb3557b5496f76f8e06fe3"
server_key_version: "encryptionkey@0"
I20260808 15:48:16.825176 28949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "75785750308649209a96fc5f2e61eb8c"
format_stamp: "Formatted at 2026-08-08 15:48:16 on dist-test-slave-twwt"
server_key: "4606199410ce6f7036ed1f691e466905"
server_key_iv: "110763c9baeb3557b5496f76f8e06fe3"
server_key_version: "encryptionkey@0"
I20260808 15:48:16.830456 28949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:48:16.834125 28979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:16.835613 28949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:16.835810 28949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "75785750308649209a96fc5f2e61eb8c"
format_stamp: "Formatted at 2026-08-08 15:48:16 on dist-test-slave-twwt"
server_key: "4606199410ce6f7036ed1f691e466905"
server_key_iv: "110763c9baeb3557b5496f76f8e06fe3"
server_key_version: "encryptionkey@0"
I20260808 15:48:16.836032 28949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:16.863095 28949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:16.868100 28949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:16.868443 28949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:16.869473 28949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:16.871008 28949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:16.871119 28949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:16.871280 28949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:16.871359 28949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:16.906875 28949 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:38831
I20260808 15:48:16.906898 29156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:38831 every 8 connection(s)
I20260808 15:48:16.908314 28949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:16.910027 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28949
I20260808 15:48:16.910259 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:16.911055 27987 external_mini_cluster.cc:1428] Setting key 6c2c33be3ae4455a1cc73543346c432f
Aug 08 15:48:16 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204096, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:16.930449 29162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:16.911298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:36689 (local address 127.27.84.254:36927)
0808 15:48:16.911851 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:16.911859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:16.913190 (+ 1331us) server_negotiation.cc:407] Connection header received
0808 15:48:16.915196 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:16.915204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:16.915307 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:16.915430 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:16.916898 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:16.918077 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:16.919717 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:16.920228 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:16.920338 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:16.924569 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:16.924574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:16.924577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:16.924608 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:16.927231 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:16.928004 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:16.928009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:16.928011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:16.928138 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:16.928590 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:16.928595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:16.928600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:16.928876 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:16.929035 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:16.929783 (+ 748us) server_negotiation.cc:299] Negotiation successful
0808 15:48:16.930036 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":274,"threads_started":1}
I20260808 15:48:16.931543 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:16.912080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36927 (local address 127.27.84.193:36689)
0808 15:48:16.913001 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:16.913072 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:16.914929 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:16.915621 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:16.915630 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:16.916148 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:16.916720 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:16.916737 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:16.918244 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:16.918248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:16.919566 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:16.919583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:16.920478 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:16.920487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:16.920823 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:16.921575 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:16.921608 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:16.924336 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:16.927394 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:16.927404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:16.927420 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:16.927863 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:16.928254 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:16.928260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:16.928263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:16.928463 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:16.929011 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:16.929020 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:16.929386 (+ 366us) client_negotiation.cc:769] Sending connection context
0808 15:48:16.929665 (+ 279us) client_negotiation.cc:240] Negotiation successful
0808 15:48:16.930264 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
I20260808 15:48:16.933369 29158 heartbeater.cc:347] Connected to a master server at 127.27.84.254:36927
I20260808 15:48:16.933673 29158 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:16.934469 29158 heartbeater.cc:511] Master 127.27.84.254:36927 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
I20260808 15:48:16.937090 28859 ts_manager.cc:194] Registered new tserver with Master: 75785750308649209a96fc5f2e61eb8c (127.27.84.193:38831)
I20260808 15:48:16.938656 28859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:36689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:16.961601 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36927
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:17.311303 29172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:17.311667 29172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:17.311867 29172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:17.319799 29172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:17.319940 29172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:17.320122 29172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:17.329962 29172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:17.331389 29172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:17.332803 29172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:17.346796 29180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:17.346725 29181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:17.347287 29183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:17.347534 29172 server_base.cc:1034] running on GCE node
I20260808 15:48:17.348179 29172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:17.349113 29172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:17.350382 29172 hybrid_clock.cc:648] HybridClock initialized: now 1786204097350329 us; error 47 us; skew 500 ppm
I20260808 15:48:17.350780 29172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:17 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204097, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:17.354244 29172 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:17.355824 29172 webserver.cc:460] Webserver started at http://127.27.84.194:38909/ using document root <none> and password file <none>
I20260808 15:48:17.356555 29172 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:17.356709 29172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:17.357048 29172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:17.360652 29172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "cc41743d794c415cb79d6dcf673a696f"
format_stamp: "Formatted at 2026-08-08 15:48:17 on dist-test-slave-twwt"
server_key: "8c43cfd5c0d0a3091f9cdfd92ce4381c"
server_key_iv: "396910202b1b80721b87f3b293146141"
server_key_version: "encryptionkey@0"
I20260808 15:48:17.361572 29172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "cc41743d794c415cb79d6dcf673a696f"
format_stamp: "Formatted at 2026-08-08 15:48:17 on dist-test-slave-twwt"
server_key: "8c43cfd5c0d0a3091f9cdfd92ce4381c"
server_key_iv: "396910202b1b80721b87f3b293146141"
server_key_version: "encryptionkey@0"
I20260808 15:48:17.367692 29172 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 15:48:17.371709 29199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:17.373286 29172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:48:17.373499 29172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "cc41743d794c415cb79d6dcf673a696f"
format_stamp: "Formatted at 2026-08-08 15:48:17 on dist-test-slave-twwt"
server_key: "8c43cfd5c0d0a3091f9cdfd92ce4381c"
server_key_iv: "396910202b1b80721b87f3b293146141"
server_key_version: "encryptionkey@0"
I20260808 15:48:17.373744 29172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:17.406445 29172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:17.411257 29172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:17.411674 29172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:17.412720 29172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:17.414156 29172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:17.414234 29172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:17.414330 29172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:17.414383 29172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:17.450937 29172 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:42863
I20260808 15:48:17.451028 29371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:42863 every 8 connection(s)
I20260808 15:48:17.452394 29172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:17.452689 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29172
I20260808 15:48:17.452908 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:17.453585 27987 external_mini_cluster.cc:1428] Setting key a669e5ffeafa892335b6f5f306ce1236
Aug 08 15:48:17 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204097, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
I20260808 15:48:17.477404 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:17.455542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:35059 (local address 127.27.84.254:36927)
0808 15:48:17.456069 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:17.456080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:17.457529 (+ 1449us) server_negotiation.cc:407] Connection header received
0808 15:48:17.459672 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:17.459680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:17.459776 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:17.459902 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:17.462539 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:17.463369 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:17.466328 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:17.467081 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:17.467257 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:17.471237 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:17.471248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:17.471252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:17.471286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:17.474038 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:17.474880 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:17.474886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:17.474890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:17.475028 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:17.475483 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:17.475487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:17.475491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:17.475773 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:17.475930 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:17.476748 (+ 818us) server_negotiation.cc:299] Negotiation successful
0808 15:48:17.477010 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":310,"threads_started":1}
I20260808 15:48:17.478569 29379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:17.456376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36927 (local address 127.27.84.194:35059)
0808 15:48:17.457335 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:17.457389 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:17.459387 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:17.460156 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:17.460164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:17.461137 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:17.462283 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:17.462335 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:17.463575 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:17.463583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:17.466153 (+ 2570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:17.466174 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:17.467448 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:17.467460 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:17.467847 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:17.468647 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:17.468670 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:17.470996 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:17.474197 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:17.474205 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:17.474222 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:17.474759 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:17.475144 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:17.475149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:17.475153 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:17.475358 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:17.475906 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:17.475914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:17.476324 (+ 410us) client_negotiation.cc:769] Sending connection context
0808 15:48:17.476631 (+ 307us) client_negotiation.cc:240] Negotiation successful
0808 15:48:17.477356 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":427,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
I20260808 15:48:17.480357 29374 heartbeater.cc:347] Connected to a master server at 127.27.84.254:36927
I20260808 15:48:17.480756 29374 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:17.481580 29374 heartbeater.cc:511] Master 127.27.84.254:36927 requested a full tablet report, sending...
I20260808 15:48:17.483323 28859 ts_manager.cc:194] Registered new tserver with Master: cc41743d794c415cb79d6dcf673a696f (127.27.84.194:42863)
I20260808 15:48:17.484231 28859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:35059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:17.507193 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36927
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:17.852743 29382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:17.853048 29382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:17.853214 29382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:17.860749 29382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:17.860888 29382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:17.861089 29382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:17.870961 29382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36927
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:17.872313 29382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:17.873716 29382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:17.887710 29393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:17.887594 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:17.888051 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:17.888558 29382 server_base.cc:1034] running on GCE node
I20260808 15:48:17.889192 29382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:17.890267 29382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:17.891507 29382 hybrid_clock.cc:648] HybridClock initialized: now 1786204097891444 us; error 55 us; skew 500 ppm
I20260808 15:48:17.891865 29382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:17 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204097, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:17.895136 29382 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:17.896703 29382 webserver.cc:460] Webserver started at http://127.27.84.195:43415/ using document root <none> and password file <none>
I20260808 15:48:17.897470 29382 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:17.897626 29382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:17.897975 29382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:17.901275 29382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "049003ddb0404c2d86667c4c4c59afcb"
format_stamp: "Formatted at 2026-08-08 15:48:17 on dist-test-slave-twwt"
server_key: "cce711abe3503f2716f1366ac40b7bda"
server_key_iv: "beb8353b29d6b633a9e29ba0567f0ea1"
server_key_version: "encryptionkey@0"
I20260808 15:48:17.902117 29382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "049003ddb0404c2d86667c4c4c59afcb"
format_stamp: "Formatted at 2026-08-08 15:48:17 on dist-test-slave-twwt"
server_key: "cce711abe3503f2716f1366ac40b7bda"
server_key_iv: "beb8353b29d6b633a9e29ba0567f0ea1"
server_key_version: "encryptionkey@0"
I20260808 15:48:17.908218 29382 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260808 15:48:17.911991 29407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:17.913468 29382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:17.913611 29382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "049003ddb0404c2d86667c4c4c59afcb"
format_stamp: "Formatted at 2026-08-08 15:48:17 on dist-test-slave-twwt"
server_key: "cce711abe3503f2716f1366ac40b7bda"
server_key_iv: "beb8353b29d6b633a9e29ba0567f0ea1"
server_key_version: "encryptionkey@0"
I20260808 15:48:17.913784 29382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:17.940402 29382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:17.942098 29158 heartbeater.cc:503] Master 127.27.84.254:36927 was elected leader, sending a full tablet report...
I20260808 15:48:17.945488 29382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:17.945750 29382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:17.946732 29382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:17.948088 29382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:17.948168 29382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:17.948253 29382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:17.948299 29382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:17.987191 29382 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:41371
I20260808 15:48:17.987296 29570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:41371 every 8 connection(s)
I20260808 15:48:17.988643 29382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:17.989722 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29382
I20260808 15:48:17.989924 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:17.990443 27987 external_mini_cluster.cc:1428] Setting key e6cd3b81c97a150d3cdb1c40ee2151f0
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204097, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:18.011925 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:17.991629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:36279 (local address 127.27.84.254:36927)
0808 15:48:17.992190 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:17.992194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:17.993214 (+ 1020us) server_negotiation.cc:407] Connection header received
0808 15:48:17.995504 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:17.995509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:17.995606 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:17.995721 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:17.997304 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:17.998092 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:17.999860 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.000350 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.000493 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.004458 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.004463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.004467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.004504 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.008263 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.009142 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.009148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.009152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.009311 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.009821 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.009826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.009829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.010145 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.010336 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.011238 (+ 902us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.011518 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":307,"threads_started":1}
I20260808 15:48:18.013096 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:17.992068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36927 (local address 127.27.84.195:36279)
0808 15:48:17.993043 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:17.993097 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:17.995238 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:17.995979 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:17.995988 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:17.996523 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:17.997118 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:17.997136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:17.998255 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:17.998261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:17.999697 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:17.999715 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.000602 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.000610 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.000870 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.001625 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:18.001648 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:18.004169 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:18.008438 (+ 4269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:18.008448 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:18.008476 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:18.008991 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:18.009435 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:18.009441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:18.009445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:18.009678 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:18.010295 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:18.010304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:18.010798 (+ 494us) client_negotiation.cc:769] Sending connection context
0808 15:48:18.011123 (+ 325us) client_negotiation.cc:240] Negotiation successful
0808 15:48:18.011773 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":355,"threads_started":1}
I20260808 15:48:18.014798 29571 heartbeater.cc:347] Connected to a master server at 127.27.84.254:36927
I20260808 15:48:18.015154 29571 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:18.015982 29571 heartbeater.cc:511] Master 127.27.84.254:36927 requested a full tablet report, sending...
I20260808 15:48:18.017546 28859 ts_manager.cc:194] Registered new tserver with Master: 049003ddb0404c2d86667c4c4c59afcb (127.27.84.195:41371)
I20260808 15:48:18.018692 28859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:36279
I20260808 15:48:18.020238 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:48:18.040241 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.026874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.27.84.254:36927)
0808 15:48:18.027060 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.027065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.027645 (+ 580us) server_negotiation.cc:407] Connection header received
0808 15:48:18.027820 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.027824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.027916 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.028007 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.029502 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.030256 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.031581 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.032013 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.032115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.034505 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.034510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.034513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.034545 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.037299 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.038019 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.038023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.038025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.038157 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.038566 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.038574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.038579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.039027 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.039222 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.039625 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.039891 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:48:18.047248 28859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:18.054204 28859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:18.080168 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.067841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.27.84.195:41371)
0808 15:48:18.070374 (+ 2533us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.070384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.070421 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:48:18.070548 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.070558 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.070845 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.071063 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:18.071809 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.073969 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.077124 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.078679 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.078848 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.079098 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.079578 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.079752 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":2125,"threads_started":1}
I20260808 15:48:18.080189 29600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.067268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:41371 (local address 127.0.0.1:59554)
0808 15:48:18.068866 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:18.068928 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:18.069141 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:18.071078 (+ 1937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:18.071089 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:18.071121 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:18.071565 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.071577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.074124 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.074131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.076923 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.076939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.078988 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.078998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.079211 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.079365 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 15:48:18.079504 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 15:48:18.079693 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"mutex_wait_us":101,"thread_start_us":1102,"threads_started":1}
I20260808 15:48:18.080461 29602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.067346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42863 (local address 127.0.0.1:48476)
0808 15:48:18.070398 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:18.070413 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:18.070571 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:18.071541 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:18.071546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:18.071570 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:18.072098 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.072112 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.073891 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.073900 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.077026 (+ 3126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.077040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.079367 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.079375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.079550 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.079579 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:48:18.079654 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 15:48:18.079784 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2942,"mutex_wait_us":300,"thread_start_us":2686,"threads_started":1}
I20260808 15:48:18.081601 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.067355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.27.84.194:42863)
0808 15:48:18.069758 (+ 2403us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.069770 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.070733 (+ 963us) server_negotiation.cc:407] Connection header received
0808 15:48:18.070826 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.070831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.071048 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.071250 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:18.072385 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.073756 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.077212 (+ 3456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.078718 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.078991 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.079190 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.080191 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.080956 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2218,"thread_start_us":2117,"threads_started":1}
I20260808 15:48:18.081934 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.070850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.27.84.193:38831)
0808 15:48:18.071900 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.071910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.072991 (+ 1081us) server_negotiation.cc:407] Connection header received
0808 15:48:18.073289 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.073297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.073501 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.073665 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:18.074754 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.075778 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.078649 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.080502 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.080627 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.080855 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.081436 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.081606 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":674,"threads_started":1}
I20260808 15:48:18.082692 29605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.070584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:38831 (local address 127.0.0.1:35388)
0808 15:48:18.072872 (+ 2288us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:18.072891 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:18.073101 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:18.073965 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:18.073969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:18.073995 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:18.074537 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.074547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.075966 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.075971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.078403 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.078420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.081016 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.081025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.081220 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.081253 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:48:18.082147 (+ 894us) client_negotiation.cc:240] Negotiation successful
0808 15:48:18.082295 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2099,"mutex_wait_us":77,"thread_start_us":1539,"threads_started":1}
I20260808 15:48:18.084812 29480 tablet_service.cc:1467] Processing CreateTablet for tablet 96fafc4e8b944003891bbe4e96d527f2 (DEFAULT_TABLE table=test-table [id=df31f291be8045869c0e6f397f8d7059]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:18.085230 29273 tablet_service.cc:1467] Processing CreateTablet for tablet 96fafc4e8b944003891bbe4e96d527f2 (DEFAULT_TABLE table=test-table [id=df31f291be8045869c0e6f397f8d7059]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:18.086483 29480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fafc4e8b944003891bbe4e96d527f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:18.087020 29273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fafc4e8b944003891bbe4e96d527f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:18.086525 29050 tablet_service.cc:1467] Processing CreateTablet for tablet 96fafc4e8b944003891bbe4e96d527f2 (DEFAULT_TABLE table=test-table [id=df31f291be8045869c0e6f397f8d7059]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:18.088411 29050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96fafc4e8b944003891bbe4e96d527f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:18.098779 29606 tablet_bootstrap.cc:492] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: Bootstrap starting.
I20260808 15:48:18.102505 29607 tablet_bootstrap.cc:492] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: Bootstrap starting.
I20260808 15:48:18.103099 29606 tablet_bootstrap.cc:654] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:18.104185 29608 tablet_bootstrap.cc:492] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: Bootstrap starting.
I20260808 15:48:18.104552 29606 log.cc:824] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:18.106451 29607 tablet_bootstrap.cc:654] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:18.107844 29607 log.cc:824] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:18.107909 29608 tablet_bootstrap.cc:654] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:18.108336 29606 tablet_bootstrap.cc:492] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: No bootstrap required, opened a new log
I20260808 15:48:18.108695 29606 ts_tablet_manager.cc:1397] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260808 15:48:18.109139 29608 log.cc:824] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:18.111238 29607 tablet_bootstrap.cc:492] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: No bootstrap required, opened a new log
I20260808 15:48:18.111505 29607 ts_tablet_manager.cc:1397] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260808 15:48:18.115193 29608 tablet_bootstrap.cc:492] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: No bootstrap required, opened a new log
I20260808 15:48:18.115500 29608 ts_tablet_manager.cc:1397] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260808 15:48:18.121086 29606 raft_consensus.cc:348] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.121572 29606 raft_consensus.cc:374] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:18.121737 29606 raft_consensus.cc:729] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049003ddb0404c2d86667c4c4c59afcb, State: Initialized, Role: FOLLOWER
I20260808 15:48:18.122551 29606 consensus_queue.cc:260] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.122769 29607 raft_consensus.cc:348] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.123289 29607 raft_consensus.cc:374] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:18.123425 29607 raft_consensus.cc:729] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc41743d794c415cb79d6dcf673a696f, State: Initialized, Role: FOLLOWER
I20260808 15:48:18.124233 29607 consensus_queue.cc:260] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.124605 29608 raft_consensus.cc:348] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.125075 29608 raft_consensus.cc:374] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:18.125198 29608 raft_consensus.cc:729] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75785750308649209a96fc5f2e61eb8c, State: Initialized, Role: FOLLOWER
I20260808 15:48:18.125784 29374 heartbeater.cc:503] Master 127.27.84.254:36927 was elected leader, sending a full tablet report...
I20260808 15:48:18.125902 29608 consensus_queue.cc:260] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.126540 29607 ts_tablet_manager.cc:1428] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260808 15:48:18.127823 29608 ts_tablet_manager.cc:1428] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 15:48:18.129031 29571 heartbeater.cc:503] Master 127.27.84.254:36927 was elected leader, sending a full tablet report...
I20260808 15:48:18.129801 29606 ts_tablet_manager.cc:1428] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
W20260808 15:48:18.160972 29161 tablet.cc:2369] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:48:18.204782 29376 tablet.cc:2369] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:48:18.240492 29572 tablet.cc:2369] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:18.311000 29616 raft_consensus.cc:482] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:18.311371 29616 raft_consensus.cc:504] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.313160 29616 leader_election.cc:288] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049003ddb0404c2d86667c4c4c59afcb (127.27.84.195:41371), 75785750308649209a96fc5f2e61eb8c (127.27.84.193:38831)
I20260808 15:48:18.320828 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.313665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:35825 (local address 127.27.84.195:41371)
0808 15:48:18.313936 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.313946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.314833 (+ 887us) server_negotiation.cc:407] Connection header received
0808 15:48:18.314985 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.314992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.315077 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.315173 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:18.315947 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.316733 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.318829 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.319630 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.319756 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.319809 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.320417 (+ 608us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.320554 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:48:18.320863 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.313897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36019 (local address 127.27.84.193:38831)
0808 15:48:18.314154 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.314163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.314995 (+ 832us) server_negotiation.cc:407] Connection header received
0808 15:48:18.315222 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.315228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.315313 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.315430 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:18.316067 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.316765 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.318918 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.319783 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.319962 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.320048 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.320480 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.320612 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:48:18.321167 29621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.313453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:41371 (local address 127.27.84.194:35825)
0808 15:48:18.314701 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:18.314725 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:18.314862 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:18.315222 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:18.315241 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:18.315268 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:18.315753 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.315765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.316896 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.316901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.318666 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.318691 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.319918 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.319927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.320227 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.320259 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:48:18.320346 (+ 87us) client_negotiation.cc:240] Negotiation successful
0808 15:48:18.320510 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1078,"mutex_wait_us":2,"thread_start_us":989,"threads_started":1}
I20260808 15:48:18.321230 29619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.313696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:38831 (local address 127.27.84.194:36019)
0808 15:48:18.314856 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:18.314876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:18.315013 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:18.315456 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:18.315459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:18.315481 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:18.315910 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.315922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.316988 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.316995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.318759 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:18.318777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.320000 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.320004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:18.320269 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.320293 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:48:18.320412 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 15:48:18.320538 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":127,"thread_start_us":444,"threads_started":1}
I20260808 15:48:18.322430 29509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fafc4e8b944003891bbe4e96d527f2" candidate_uuid: "cc41743d794c415cb79d6dcf673a696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049003ddb0404c2d86667c4c4c59afcb" is_pre_election: true
I20260808 15:48:18.322432 29088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fafc4e8b944003891bbe4e96d527f2" candidate_uuid: "cc41743d794c415cb79d6dcf673a696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75785750308649209a96fc5f2e61eb8c" is_pre_election: true
I20260808 15:48:18.323091 29509 raft_consensus.cc:2454] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc41743d794c415cb79d6dcf673a696f in term 0.
I20260808 15:48:18.323091 29088 raft_consensus.cc:2454] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc41743d794c415cb79d6dcf673a696f in term 0.
I20260808 15:48:18.323887 29201 leader_election.cc:302] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049003ddb0404c2d86667c4c4c59afcb, cc41743d794c415cb79d6dcf673a696f; no voters:
I20260808 15:48:18.324376 29616 raft_consensus.cc:2781] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:18.324540 29616 raft_consensus.cc:482] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:18.324669 29616 raft_consensus.cc:3037] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:18.327014 29616 raft_consensus.cc:504] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.327674 29616 leader_election.cc:288] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [CANDIDATE]: Term 1 election: Requested vote from peers 049003ddb0404c2d86667c4c4c59afcb (127.27.84.195:41371), 75785750308649209a96fc5f2e61eb8c (127.27.84.193:38831)
I20260808 15:48:18.328228 29509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fafc4e8b944003891bbe4e96d527f2" candidate_uuid: "cc41743d794c415cb79d6dcf673a696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049003ddb0404c2d86667c4c4c59afcb"
I20260808 15:48:18.328240 29088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96fafc4e8b944003891bbe4e96d527f2" candidate_uuid: "cc41743d794c415cb79d6dcf673a696f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75785750308649209a96fc5f2e61eb8c"
I20260808 15:48:18.328444 29509 raft_consensus.cc:3037] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:18.328534 29088 raft_consensus.cc:3037] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:18.330869 29509 raft_consensus.cc:2454] T 96fafc4e8b944003891bbe4e96d527f2 P 049003ddb0404c2d86667c4c4c59afcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc41743d794c415cb79d6dcf673a696f in term 1.
I20260808 15:48:18.330933 29088 raft_consensus.cc:2454] T 96fafc4e8b944003891bbe4e96d527f2 P 75785750308649209a96fc5f2e61eb8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc41743d794c415cb79d6dcf673a696f in term 1.
I20260808 15:48:18.331321 29201 leader_election.cc:302] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049003ddb0404c2d86667c4c4c59afcb, cc41743d794c415cb79d6dcf673a696f; no voters:
I20260808 15:48:18.331660 29616 raft_consensus.cc:2781] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:18.332152 29616 raft_consensus.cc:686] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [term 1 LEADER]: Becoming Leader. State: Replica: cc41743d794c415cb79d6dcf673a696f, State: Running, Role: LEADER
I20260808 15:48:18.332657 29616 consensus_queue.cc:237] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } }
I20260808 15:48:18.338055 28858 catalog_manager.cc:5495] T 96fafc4e8b944003891bbe4e96d527f2 P cc41743d794c415cb79d6dcf673a696f reported cstate change: term changed from 0 to 1, leader changed from <none> to cc41743d794c415cb79d6dcf673a696f (127.27.84.194). New cstate: current_term: 1 leader_uuid: "cc41743d794c415cb79d6dcf673a696f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc41743d794c415cb79d6dcf673a696f" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "049003ddb0404c2d86667c4c4c59afcb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75785750308649209a96fc5f2e61eb8c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38831 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:18.384251 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.368441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35404 (local address 127.27.84.193:38831)
0808 15:48:18.368635 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.368639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.369236 (+ 597us) server_negotiation.cc:407] Connection header received
0808 15:48:18.369325 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.369330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.369411 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.369511 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.371167 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.371938 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.373481 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.373929 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.374033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.378171 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.378205 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.378222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.378304 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.381512 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.382214 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.382222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.382224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.382391 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.382904 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.382907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.382909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.383281 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.383529 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.383806 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.383976 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204098, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:18.445330 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.430850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.27.84.193:38831)
0808 15:48:18.431097 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.431106 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.431462 (+ 356us) server_negotiation.cc:407] Connection header received
0808 15:48:18.431695 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.431700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.431769 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.431882 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.433346 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.434094 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.435442 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.435865 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.436010 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.440110 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.440116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.440120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.440157 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.443109 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.443685 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.443689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.443692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.443822 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.444175 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.444178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.444180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.444460 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.444667 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.444887 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.445052 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:18.499166 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.484335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.27.84.193:38831)
0808 15:48:18.484598 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.484606 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.484849 (+ 243us) server_negotiation.cc:407] Connection header received
0808 15:48:18.485085 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.485089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.485156 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.485265 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.486708 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.487440 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.488736 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.489211 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.489353 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.493628 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.493632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.493636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.493679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.496402 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.497040 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.497047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.497050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.497177 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.497555 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.497560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.497564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.497885 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.498151 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.498461 (+ 310us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.498755 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260808 15:48:18.501506 29020 tablet_service.cc:2908] Rejecting scan request for tablet 96fafc4e8b944003891bbe4e96d527f2: Uninitialized: clean time has not yet been initialized
I20260808 15:48:18.505370 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28949
I20260808 15:48:18.518366 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.506383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.27.84.193:38831)
0808 15:48:18.506542 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.506546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.507066 (+ 520us) server_negotiation.cc:407] Connection header received
0808 15:48:18.507211 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.507215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.507290 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.507372 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.508534 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.509214 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.510447 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.510951 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.511062 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.513319 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.513325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.513328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.513362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.515980 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.516659 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.516663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.516666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.516788 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.517151 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.517154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.517156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.517518 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.517814 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.517967 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.518133 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:48:18.519117 29149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:18.610443 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28949
I20260808 15:48:18.632645 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29172
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:18.647712 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.633522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.27.84.194:42863)
0808 15:48:18.633968 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.633973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.633989 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:48:18.634046 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.634050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.634107 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.634196 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.635541 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.636232 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.637465 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.637866 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.637959 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.642047 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.642061 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.642077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.642110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.644786 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.645490 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.645499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.645501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.645648 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.646023 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.646028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.646031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.646380 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.646711 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.647102 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.647348 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":217,"threads_started":1}
I20260808 15:48:18.648453 29363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:18.739210 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29172
I20260808 15:48:18.766923 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29382
Aug 08 15:48:18 dist-test-slave-twwt krb5kdc[28785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:18.782229 29603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:18.767727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.27.84.195:41371)
0808 15:48:18.767942 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:18.767950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:18.767971 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:48:18.768188 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:18.768198 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:18.768269 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:18.768391 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:18.769724 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.770488 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.771856 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:18.772259 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:18.772404 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:18.776718 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:18.776735 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:18.776752 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:18.776794 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:18.779687 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.780320 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.780327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.780330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.780460 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:18.780857 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:18.780861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:18.780864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:18.781244 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:18.781496 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:18.781787 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:48:18.781966 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:48:18.783051 29563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:18.871918 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29382
I20260808 15:48:18.904321 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28806
I20260808 15:48:18.905592 28898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:19.018082 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28806
2026-08-08T15:48:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4818 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:19 dist-test-slave-twwt krb5kdc[29653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:19 dist-test-slave-twwt krb5kdc[29653](info): set up 2 sockets
Aug 08 15:48:19 dist-test-slave-twwt krb5kdc[29653](info): commencing operation
krb5kdc: starting...
W20260808 15:48:20.367851 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:20 dist-test-slave-twwt krb5kdc[29653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:20.594914 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42895
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:20.942953 29674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:20.943333 29674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:20.943500 29674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:20.951007 29674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:20.951162 29674 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:48:20.951256 29674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:20.951360 29674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:20.951449 29674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:20.951560 29674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:20.961853 29674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:20.963315 29674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:20.964720 29674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:20.977360 29686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:20.977747 29685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:20.977314 29690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:20.978826 29674 server_base.cc:1034] running on GCE node
I20260808 15:48:20.979724 29674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:20.981421 29674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:20.982801 29674 hybrid_clock.cc:648] HybridClock initialized: now 1786204100982678 us; error 47 us; skew 500 ppm
I20260808 15:48:20.983193 29674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:20 dist-test-slave-twwt krb5kdc[29653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204100, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:20.986900 29674 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:20.988575 29674 webserver.cc:460] Webserver started at http://127.27.84.254:33553/ using document root <none> and password file <none>
I20260808 15:48:20.989339 29674 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:20.989477 29674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:20.989833 29674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:20.992995 29674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0"
format_stamp: "Formatted at 2026-08-08 15:48:20 on dist-test-slave-twwt"
server_key: "18d7115cb1d9127008430f963ebab135"
server_key_iv: "1aad3b7bf46834e679715ff078c2aab3"
server_key_version: "encryptionkey@0"
I20260808 15:48:20.993819 29674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0"
format_stamp: "Formatted at 2026-08-08 15:48:20 on dist-test-slave-twwt"
server_key: "18d7115cb1d9127008430f963ebab135"
server_key_iv: "1aad3b7bf46834e679715ff078c2aab3"
server_key_version: "encryptionkey@0"
I20260808 15:48:20.999408 29674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:48:21.003105 29700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:21.004665 29674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:48:21.004900 29674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0"
format_stamp: "Formatted at 2026-08-08 15:48:20 on dist-test-slave-twwt"
server_key: "18d7115cb1d9127008430f963ebab135"
server_key_iv: "1aad3b7bf46834e679715ff078c2aab3"
server_key_version: "encryptionkey@0"
I20260808 15:48:21.005155 29674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:21.037079 29674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:21.041962 29674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:21.042276 29674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:21.065500 29674 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:42895
I20260808 15:48:21.065522 29790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:42895 every 8 connection(s)
I20260808 15:48:21.067051 29674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:21.068593 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29674
I20260808 15:48:21.068861 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:21.069703 27987 external_mini_cluster.cc:1428] Setting key 32fd3b769bf3385a226925bc14909b1f
I20260808 15:48:21.072561 29791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:48:21 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:21.087090 29791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: Bootstrap starting.
I20260808 15:48:21.091601 29791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:21.092748 29791 log.cc:824] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:21.095865 29795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:21.072063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.27.84.254:42895)
0808 15:48:21.072995 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:21.073008 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:21.073052 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:48:21.074820 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:21.074849 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:21.075284 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:21.075656 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:21.077529 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.079072 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.080882 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.081609 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.081754 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:21.086962 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:21.086974 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:21.086995 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:21.087044 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:21.091069 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:21.091822 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:21.091836 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:21.091849 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:21.092050 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:21.092489 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:21.092494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:21.092497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:21.093160 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:21.093463 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:21.093971 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 15:48:21.094412 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":404,"threads_started":1}
I20260808 15:48:21.096626 29791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: No bootstrap required, opened a new log
I20260808 15:48:21.104154 29791 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42895 } }
I20260808 15:48:21.104583 29791 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:21.104725 29791 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7a2baefdb44fb4bc8468a0d1ca66f0, State: Initialized, Role: FOLLOWER
I20260808 15:48:21.105372 29791 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42895 } }
I20260808 15:48:21.105594 29791 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:21.105693 29791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:21.105835 29791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:21.108048 29791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42895 } }
I20260808 15:48:21.108618 29791 leader_election.cc:302] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f7a2baefdb44fb4bc8468a0d1ca66f0; no voters:
I20260808 15:48:21.109289 29791 leader_election.cc:288] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:21.109403 29797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:21.110882 29797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f7a2baefdb44fb4bc8468a0d1ca66f0, State: Running, Role: LEADER
I20260808 15:48:21.111508 29797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42895 } }
I20260808 15:48:21.111922 29791 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:21.114778 29798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42895 } } }
I20260808 15:48:21.114962 29800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f7a2baefdb44fb4bc8468a0d1ca66f0. Latest consensus state: current_term: 1 leader_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f7a2baefdb44fb4bc8468a0d1ca66f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42895 } } }
I20260808 15:48:21.115242 29798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:21.115324 29800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:21.117959 29808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:21.124272 29808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:21.135267 29808 catalog_manager.cc:1347] Generated new cluster ID: 8386fc1933f64ce59bfd36efe1f5f2d8
I20260808 15:48:21.135479 29808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:21.170097 29808 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:21.172173 29808 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:21.178833 29808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: Generated new TSK 0
I20260808 15:48:21.179747 29808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:21.244297 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42895
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:21.607318 29827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:21.607628 29827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:21.607728 29827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:21.615857 29827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:21.615983 29827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:21.616070 29827 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:48:21.616155 29827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:21.628141 29827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:21.629614 29827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:21.631065 29827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:21.644820 29838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:21.644882 29836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:21.644863 29840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:21.645592 29827 server_base.cc:1034] running on GCE node
I20260808 15:48:21.646271 29827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:21.647260 29827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:21.648536 29827 hybrid_clock.cc:648] HybridClock initialized: now 1786204101648461 us; error 50 us; skew 500 ppm
I20260808 15:48:21.648870 29827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:21 dist-test-slave-twwt krb5kdc[29653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:21.657770 29827 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:21.659483 29827 webserver.cc:460] Webserver started at http://127.27.84.193:39559/ using document root <none> and password file <none>
I20260808 15:48:21.660240 29827 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:21.660386 29827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:21.660753 29827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:21.663887 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "16f1d91ddd4742fc97ee743d67dce33c"
format_stamp: "Formatted at 2026-08-08 15:48:21 on dist-test-slave-twwt"
server_key: "6b56843067436d744741be7a96bee26d"
server_key_iv: "31f015586c9dc34887e1ce9871a8cdf3"
server_key_version: "encryptionkey@0"
I20260808 15:48:21.664752 29827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "16f1d91ddd4742fc97ee743d67dce33c"
format_stamp: "Formatted at 2026-08-08 15:48:21 on dist-test-slave-twwt"
server_key: "6b56843067436d744741be7a96bee26d"
server_key_iv: "31f015586c9dc34887e1ce9871a8cdf3"
server_key_version: "encryptionkey@0"
I20260808 15:48:21.670331 29827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:48:21.674211 29848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:21.675779 29827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:21.675995 29827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "16f1d91ddd4742fc97ee743d67dce33c"
format_stamp: "Formatted at 2026-08-08 15:48:21 on dist-test-slave-twwt"
server_key: "6b56843067436d744741be7a96bee26d"
server_key_iv: "31f015586c9dc34887e1ce9871a8cdf3"
server_key_version: "encryptionkey@0"
I20260808 15:48:21.676251 29827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:21.690793 29827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:21.695536 29827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:21.695840 29827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:21.698720 29827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:21.698833 29827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:21.699018 29827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:21.699103 29827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:48:21 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204101, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:21.725595 29866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:21.699808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:56073 (local address 127.27.84.254:42895)
0808 15:48:21.702370 (+ 2562us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:21.702379 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:21.702407 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:48:21.704158 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:21.704164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:21.704258 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:21.704401 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:21.706861 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.708032 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.711562 (+ 3530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.712038 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.712205 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:21.717190 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:21.717197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:21.717202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:21.717255 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:21.721493 (+ 4238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:21.722702 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:21.722713 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:21.722719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:21.722928 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:21.723552 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:21.723559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:21.723565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:21.723951 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:21.724125 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:21.724951 (+ 826us) server_negotiation.cc:299] Negotiation successful
0808 15:48:21.725190 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2387,"thread_start_us":2270,"threads_started":1}
I20260808 15:48:21.726835 29867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:21.700468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.27.84.193:56073)
0808 15:48:21.701659 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:21.701732 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:21.703844 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:21.704649 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:21.704660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:21.705444 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:21.706550 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:21.706586 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.708255 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.708261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:21.711344 (+ 3083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:21.711380 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.712356 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.712365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:21.712582 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:21.713543 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:21.713568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:21.716795 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:21.721702 (+ 4907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:21.721719 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:21.721741 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:21.722503 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:21.723067 (+ 564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:21.723078 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:21.723082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:21.723340 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:21.724119 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:21.724128 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:21.724520 (+ 392us) client_negotiation.cc:769] Sending connection context
0808 15:48:21.724833 (+ 313us) client_negotiation.cc:240] Negotiation successful
0808 15:48:21.725430 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":455,"threads_started":1}
I20260808 15:48:21.749022 29971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:21.733722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.27.84.193:48641)
0808 15:48:21.734534 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:21.734560 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:21.734773 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:21.735206 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:21.735211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:21.735766 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:21.736263 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:21.736281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.737351 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.737356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:21.739029 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:21.739043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.739834 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.739844 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:21.740053 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:21.740929 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:21.740961 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:21.742535 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:21.746031 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:21.746037 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:21.746041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:21.746469 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:21.746913 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:21.746922 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:21.746925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:21.747185 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:21.748088 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:21.748092 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:21.748265 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 15:48:21.748386 (+ 121us) client_negotiation.cc:240] Negotiation successful
0808 15:48:21.748649 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":415,"threads_started":1}
I20260808 15:48:21.749413 29866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:21.733744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:48641 (local address 127.27.84.254:42895)
0808 15:48:21.733983 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:21.733991 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:21.734680 (+ 689us) server_negotiation.cc:407] Connection header received
0808 15:48:21.734936 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:21.734943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:21.735035 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:21.735179 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:21.736469 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.737173 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.739193 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:21.739625 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:21.739763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:21.742807 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:21.742816 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:21.742823 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:21.742863 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:21.745884 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:21.746624 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:21.746632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:21.746638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:21.746789 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:21.747351 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:21.747359 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:21.747365 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:21.747898 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:21.748098 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:21.748572 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 15:48:21.748902 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:48:21.759716 29827 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43277
I20260808 15:48:21.759737 30050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43277 every 8 connection(s)
I20260808 15:48:21.761117 29827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:21.761811 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29827
I20260808 15:48:21.762100 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:21.762959 27987 external_mini_cluster.cc:1428] Setting key 417cae1a4d69475e6d6b9450bc94c847
I20260808 15:48:21.763831 30051 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42895
I20260808 15:48:21.764099 30051 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:21.764863 30051 heartbeater.cc:511] Master 127.27.84.254:42895 requested a full tablet report, sending...
I20260808 15:48:21.767421 29720 ts_manager.cc:194] Registered new tserver with Master: 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277)
I20260808 15:48:21.769683 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:56073
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:21.815618 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42895
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:22.179210 30055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:22.179533 30055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:22.179729 30055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:22.187412 30055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:22.187585 30055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:22.187745 30055 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:48:22.187877 30055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:22.198408 30055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:22.199925 30055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:22.201467 30055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:22.217047 30062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:22.216977 30068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:22.217010 30065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:22.218039 30055 server_base.cc:1034] running on GCE node
I20260808 15:48:22.218827 30055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:22.220042 30055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:22.221284 30055 hybrid_clock.cc:648] HybridClock initialized: now 1786204102221205 us; error 55 us; skew 500 ppm
I20260808 15:48:22.221637 30055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:22 dist-test-slave-twwt krb5kdc[29653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:22.225349 30055 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:22.227197 30055 webserver.cc:460] Webserver started at http://127.27.84.194:42305/ using document root <none> and password file <none>
I20260808 15:48:22.228127 30055 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:22.228272 30055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:22.228644 30055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:22.232117 30055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "2e4e3b29856b409f8baba6bdd369cd58"
format_stamp: "Formatted at 2026-08-08 15:48:22 on dist-test-slave-twwt"
server_key: "6fc493cd0ae1c861bd5436ff51c80809"
server_key_iv: "34aea7ab336604c38ec618c5cc8c41fe"
server_key_version: "encryptionkey@0"
I20260808 15:48:22.233058 30055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "2e4e3b29856b409f8baba6bdd369cd58"
format_stamp: "Formatted at 2026-08-08 15:48:22 on dist-test-slave-twwt"
server_key: "6fc493cd0ae1c861bd5436ff51c80809"
server_key_iv: "34aea7ab336604c38ec618c5cc8c41fe"
server_key_version: "encryptionkey@0"
I20260808 15:48:22.239023 30055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 15:48:22.242983 30080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:22.244571 30055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:22.244809 30055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "2e4e3b29856b409f8baba6bdd369cd58"
format_stamp: "Formatted at 2026-08-08 15:48:22 on dist-test-slave-twwt"
server_key: "6fc493cd0ae1c861bd5436ff51c80809"
server_key_iv: "34aea7ab336604c38ec618c5cc8c41fe"
server_key_version: "encryptionkey@0"
I20260808 15:48:22.245080 30055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:22.263676 30055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:22.268810 30055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:22.269070 30055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:22.272228 30055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:22.272338 30055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:22.272496 30055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:22.272583 30055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:48:22 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:22.298492 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.272873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:50049 (local address 127.27.84.254:42895)
0808 15:48:22.273430 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:22.273439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:22.274757 (+ 1318us) server_negotiation.cc:407] Connection header received
0808 15:48:22.277098 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:22.277107 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:22.277219 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:22.277370 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:22.280419 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.281550 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.284564 (+ 3014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.284993 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.285155 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.291076 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:22.291082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:22.291086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:22.291128 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:22.294420 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.295684 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.295690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.295693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.295876 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.296407 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.296417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.296419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.296708 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:22.296899 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:22.297825 (+ 926us) server_negotiation.cc:299] Negotiation successful
0808 15:48:22.298078 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":279,"threads_started":1}
I20260808 15:48:22.300179 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.273609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.27.84.194:50049)
0808 15:48:22.274514 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:22.274567 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:22.276621 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:22.277704 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:22.277717 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:22.278699 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:22.280141 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.280186 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.281747 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.281753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.284292 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.284315 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.285284 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.285292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.285673 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.286670 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:22.286714 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:22.290596 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:22.294647 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.294659 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.294680 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.295506 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.296011 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.296017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.296021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.296260 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.296873 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:22.296882 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:22.297392 (+ 510us) client_negotiation.cc:769] Sending connection context
0808 15:48:22.297694 (+ 302us) client_negotiation.cc:240] Negotiation successful
0808 15:48:22.298326 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":373,"threads_started":1}
I20260808 15:48:22.328606 30055 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:45303
I20260808 15:48:22.328670 30274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:45303 every 8 connection(s)
I20260808 15:48:22.330224 30055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:22.332916 30095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.307155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:43093 (local address 127.27.84.254:42895)
0808 15:48:22.307376 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:22.307384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:22.313902 (+ 6518us) server_negotiation.cc:407] Connection header received
0808 15:48:22.314138 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:22.314144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:22.314237 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:22.314351 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:22.316523 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.317694 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.321289 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.321886 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.322015 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.325632 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:22.325642 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:22.325648 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:22.325700 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:22.329511 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.330336 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.330341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.330343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.330525 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.331109 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.331116 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.331122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.331639 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:22.331810 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:22.332295 (+ 485us) server_negotiation.cc:299] Negotiation successful
0808 15:48:22.332549 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:48:22.332917 30275 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42895
I20260808 15:48:22.333025 30209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.307139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.27.84.194:43093)
0808 15:48:22.313774 (+ 6635us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:22.313792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:22.313987 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:22.314454 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:22.314462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:22.315510 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:22.316273 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.316295 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.317861 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.317869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.321046 (+ 3177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.321069 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.322435 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.322446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.322737 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.323766 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:22.323798 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:22.325325 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:22.329662 (+ 4337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.329667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.329672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.330173 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.330689 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.330695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.330698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.330921 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.331783 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:22.331788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:22.332005 (+ 217us) client_negotiation.cc:769] Sending connection context
0808 15:48:22.332155 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 15:48:22.332541 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6489,"thread_start_us":6951,"threads_started":1}
I20260808 15:48:22.333554 30275 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:22.334720 30275 heartbeater.cc:511] Master 127.27.84.254:42895 requested a full tablet report, sending...
I20260808 15:48:22.336247 29720 ts_manager.cc:194] Registered new tserver with Master: 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303)
I20260808 15:48:22.337070 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:50049
I20260808 15:48:22.340289 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30055
I20260808 15:48:22.340468 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:22.340850 27987 external_mini_cluster.cc:1428] Setting key 45eeb9e720cbe24b977e1cd57be22223
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:22.392560 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42895
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:22.744979 30279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:22.745268 30279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:22.745369 30279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:22.752964 30279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:22.753082 30279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:22.753171 30279 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:48:22.753255 30279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:22.763162 30279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:22.764451 30279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:22.765831 30279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 15:48:22.773488 30051 heartbeater.cc:503] Master 127.27.84.254:42895 was elected leader, sending a full tablet report...
W20260808 15:48:22.780110 30289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:22.780108 30286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:22.780146 30287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:22.781174 30279 server_base.cc:1034] running on GCE node
I20260808 15:48:22.781797 30279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:22.782858 30279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:22.784099 30279 hybrid_clock.cc:648] HybridClock initialized: now 1786204102784039 us; error 28 us; skew 500 ppm
I20260808 15:48:22.784406 30279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:22 dist-test-slave-twwt krb5kdc[29653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:22.791241 30279 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:22.792914 30279 webserver.cc:460] Webserver started at http://127.27.84.195:45835/ using document root <none> and password file <none>
I20260808 15:48:22.793674 30279 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:22.793766 30279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:22.794050 30279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:22.797369 30279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "b32c676c5e9d41c89f3d3b1a18d053cb"
format_stamp: "Formatted at 2026-08-08 15:48:22 on dist-test-slave-twwt"
server_key: "1b97d4f261d142ce0ee867148f8120c2"
server_key_iv: "5ba581db5fda2bbf692383c8be016df1"
server_key_version: "encryptionkey@0"
I20260808 15:48:22.798244 30279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "b32c676c5e9d41c89f3d3b1a18d053cb"
format_stamp: "Formatted at 2026-08-08 15:48:22 on dist-test-slave-twwt"
server_key: "1b97d4f261d142ce0ee867148f8120c2"
server_key_iv: "5ba581db5fda2bbf692383c8be016df1"
server_key_version: "encryptionkey@0"
I20260808 15:48:22.803996 30279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:48:22.807664 30297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:22.809180 30279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:48:22.809369 30279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "b32c676c5e9d41c89f3d3b1a18d053cb"
format_stamp: "Formatted at 2026-08-08 15:48:22 on dist-test-slave-twwt"
server_key: "1b97d4f261d142ce0ee867148f8120c2"
server_key_iv: "5ba581db5fda2bbf692383c8be016df1"
server_key_version: "encryptionkey@0"
I20260808 15:48:22.809610 30279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:22.832126 30279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:22.836827 30279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:22.837154 30279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:22.840427 30279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:22.840601 30279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 15:48:22.840799 30279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:22.840912 30279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:48:22 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:22.876995 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.841896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.27.84.195:50675)
0808 15:48:22.843680 (+ 1784us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:22.843756 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:22.847670 (+ 3914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:22.849067 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:22.849084 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:22.850193 (+ 1109us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:22.853152 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.853185 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.856044 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.856052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.859204 (+ 3152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.859226 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.860774 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.860790 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.861235 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.862349 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:22.862399 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:22.866994 (+ 4595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:22.871604 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.871616 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.871637 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.872331 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.872835 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.872842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.872848 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.873108 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.873890 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:22.873902 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:22.874422 (+ 520us) client_negotiation.cc:769] Sending connection context
0808 15:48:22.874762 (+ 340us) client_negotiation.cc:240] Negotiation successful
0808 15:48:22.875380 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"thread_start_us":1178,"threads_started":1}
I20260808 15:48:22.878696 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.842171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:50675 (local address 127.27.84.254:42895)
0808 15:48:22.847112 (+ 4941us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:22.847121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:22.847154 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:48:22.848204 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:22.848213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:22.848406 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:22.848602 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:22.854429 (+ 5827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.855768 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.859419 (+ 3651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.860248 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.860417 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.867423 (+ 7006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:22.867429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:22.867433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:22.867473 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:22.871412 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.872501 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.872509 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.872516 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.872675 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.873252 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.873260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.873265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.873666 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:22.873813 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:22.877876 (+ 4063us) server_negotiation.cc:299] Negotiation successful
0808 15:48:22.878153 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4764,"thread_start_us":4637,"threads_started":1}
I20260808 15:48:22.906184 30388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.886167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.27.84.195:54649)
0808 15:48:22.886992 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:22.887020 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:22.887244 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:22.887771 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:22.887774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:22.888246 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:22.888547 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.888556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.890088 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.890094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.892201 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:22.892231 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.893430 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.893436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:22.893768 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.895188 (+ 1420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:22.895216 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:22.896411 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:22.902131 (+ 5720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.902136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.902141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.902893 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.903438 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:22.903444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:22.903450 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:22.903645 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:22.904570 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:22.904577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:22.904843 (+ 266us) client_negotiation.cc:769] Sending connection context
0808 15:48:22.905027 (+ 184us) client_negotiation.cc:240] Negotiation successful
0808 15:48:22.905508 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":503,"threads_started":1}
I20260808 15:48:22.907367 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.886166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:54649 (local address 127.27.84.254:42895)
0808 15:48:22.886403 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:22.886412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:22.887107 (+ 695us) server_negotiation.cc:407] Connection header received
0808 15:48:22.887443 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:22.887450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:22.887529 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:22.887588 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:22.888713 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.889924 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.892440 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.893208 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.893344 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.896681 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:22.896691 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:22.896699 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:22.896751 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:22.901950 (+ 5199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.903078 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.903084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.903087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.903268 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.903811 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.903819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.903824 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.904301 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:22.904491 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:22.906644 (+ 2153us) server_negotiation.cc:299] Negotiation successful
0808 15:48:22.906918 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:48:22.920039 30483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:42251 every 8 connection(s)
I20260808 15:48:22.920042 30279 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:42251
I20260808 15:48:22.922142 30279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:22.924713 30485 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42895
I20260808 15:48:22.924996 30485 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:22.925712 30485 heartbeater.cc:511] Master 127.27.84.254:42895 requested a full tablet report, sending...
I20260808 15:48:22.927304 29720 ts_manager.cc:194] Registered new tserver with Master: b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195:42251)
I20260808 15:48:22.928177 29720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:50675
I20260808 15:48:22.929472 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30279
I20260808 15:48:22.929697 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:22.930285 27987 external_mini_cluster.cc:1428] Setting key 31bdfed84bfb68e424c24d3ea5ab0ae8
I20260808 15:48:22.933434 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:48:22 dist-test-slave-twwt krb5kdc[29653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:48:22 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:22.989144 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:22.974198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.27.84.254:42895)
0808 15:48:22.974403 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:22.974408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:22.974861 (+ 453us) server_negotiation.cc:407] Connection header received
0808 15:48:22.974980 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:22.974984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:22.975054 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:22.975151 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:22.976749 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.977478 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.978918 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:22.979410 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:22.979510 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:22.983656 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:22.983661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:22.983664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:22.983707 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:22.986430 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.987146 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.987157 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.987160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.987293 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:22.987685 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:22.987694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:22.987696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:22.987972 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:22.988246 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:22.988564 (+ 318us) server_negotiation.cc:299] Negotiation successful
0808 15:48:22.988806 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:48:22.993292 29720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:23.000211 29720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:23.023598 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.014232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.27.84.194:45303)
0808 15:48:23.014963 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.014977 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.015677 (+ 700us) server_negotiation.cc:407] Connection header received
0808 15:48:23.016105 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.016113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.016368 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.016497 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.017228 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.018735 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.020847 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.022293 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.022483 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.022753 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.023066 (+ 313us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.023257 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":1176,"threads_started":1}
I20260808 15:48:23.023602 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.014057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:45303 (local address 127.0.0.1:49232)
0808 15:48:23.015520 (+ 1463us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.015571 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.015856 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.016730 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.016736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.016759 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.017053 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.017062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.018885 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.018889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.020667 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.020682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.022548 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.022554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.022825 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.022869 (+ 44us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.023017 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.023245 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1205,"threads_started":1}
I20260808 15:48:23.037263 30167 tablet_service.cc:1467] Processing CreateTablet for tablet eaa80fabbd114e3d87bccc93c6e2a187 (DEFAULT_TABLE table=test-table [id=057c352293444d109894c65d361fd42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:23.039093 30167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa80fabbd114e3d87bccc93c6e2a187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.040048 30512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.014465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43277 (local address 127.0.0.1:60288)
0808 15:48:23.023872 (+ 9407us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.023892 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.024001 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.032806 (+ 8805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.032811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.032850 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.033078 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.033091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.034956 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.034961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.036809 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.036829 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.039274 (+ 2445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.039280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.039427 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.039451 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.039565 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.039709 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9275,"thread_start_us":10521,"threads_started":1}
I20260808 15:48:23.040376 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.015095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.27.84.193:43277)
0808 15:48:23.032208 (+ 17113us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.032216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.032244 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:48:23.032342 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.032349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.032570 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.032830 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.033285 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.034785 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.037071 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.038552 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.038927 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.039157 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.039654 (+ 497us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.039831 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16948,"thread_start_us":16803,"threads_started":1}
I20260808 15:48:23.041983 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.015144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.27.84.195:42251)
0808 15:48:23.033223 (+ 18079us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.033233 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.033270 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:48:23.033385 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.033395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.033680 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.033902 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.034588 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.036422 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.039025 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.040473 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.040619 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.040875 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.041416 (+ 541us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.041635 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17864,"thread_start_us":17687,"threads_started":1}
I20260808 15:48:23.042367 30511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.014831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:42251 (local address 127.0.0.1:39544)
0808 15:48:23.025297 (+ 10466us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.025321 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.025459 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.033907 (+ 8448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.033912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.033943 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.034348 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.034361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.036601 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.036606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.038805 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.038821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.041013 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.041019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.041191 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.041224 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.041333 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.041479 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10316,"thread_start_us":776,"threads_started":1}
I20260808 15:48:23.044453 29935 tablet_service.cc:1467] Processing CreateTablet for tablet eaa80fabbd114e3d87bccc93c6e2a187 (DEFAULT_TABLE table=test-table [id=057c352293444d109894c65d361fd42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:23.046067 29935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa80fabbd114e3d87bccc93c6e2a187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.046337 30393 tablet_service.cc:1467] Processing CreateTablet for tablet eaa80fabbd114e3d87bccc93c6e2a187 (DEFAULT_TABLE table=test-table [id=057c352293444d109894c65d361fd42d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:23.047878 30393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa80fabbd114e3d87bccc93c6e2a187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.055015 30516 tablet_bootstrap.cc:492] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Bootstrap starting.
I20260808 15:48:23.057821 30516 tablet_bootstrap.cc:654] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.058981 30516 log.cc:824] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:23.061862 30516 tablet_bootstrap.cc:492] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: No bootstrap required, opened a new log
I20260808 15:48:23.061810 30520 tablet_bootstrap.cc:492] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Bootstrap starting.
I20260808 15:48:23.062152 30516 ts_tablet_manager.cc:1397] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 15:48:23.063395 30521 tablet_bootstrap.cc:492] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Bootstrap starting.
I20260808 15:48:23.065984 30520 tablet_bootstrap.cc:654] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.067301 30520 log.cc:824] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:23.070230 30521 tablet_bootstrap.cc:654] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.071517 30521 log.cc:824] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:23.072340 30520 tablet_bootstrap.cc:492] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: No bootstrap required, opened a new log
I20260808 15:48:23.072728 30520 ts_tablet_manager.cc:1397] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260808 15:48:23.074409 30516 raft_consensus.cc:348] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.075024 30516 raft_consensus.cc:374] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.075152 30516 raft_consensus.cc:729] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4e3b29856b409f8baba6bdd369cd58, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.076092 30516 consensus_queue.cc:260] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.079113 30275 heartbeater.cc:503] Master 127.27.84.254:42895 was elected leader, sending a full tablet report...
I20260808 15:48:23.080422 30516 ts_tablet_manager.cc:1428] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
W20260808 15:48:23.082521 30276 tablet.cc:2369] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:23.082918 30521 tablet_bootstrap.cc:492] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: No bootstrap required, opened a new log
I20260808 15:48:23.083256 30521 ts_tablet_manager.cc:1397] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.006s
I20260808 15:48:23.086133 30520 raft_consensus.cc:348] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.086673 30520 raft_consensus.cc:374] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.086786 30520 raft_consensus.cc:729] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f1d91ddd4742fc97ee743d67dce33c, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.087503 30520 consensus_queue.cc:260] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.089286 30520 ts_tablet_manager.cc:1428] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260808 15:48:23.092686 30521 raft_consensus.cc:348] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.093228 30521 raft_consensus.cc:374] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.093355 30521 raft_consensus.cc:729] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32c676c5e9d41c89f3d3b1a18d053cb, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.094017 30521 consensus_queue.cc:260] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.095206 30485 heartbeater.cc:503] Master 127.27.84.254:42895 was elected leader, sending a full tablet report...
I20260808 15:48:23.095715 30521 ts_tablet_manager.cc:1428] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
W20260808 15:48:23.174144 30486 tablet.cc:2369] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:23.244975 30530 raft_consensus.cc:482] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:23.245373 30530 raft_consensus.cc:504] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.247193 30530 leader_election.cc:288] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303), 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277)
I20260808 15:48:23.250368 30529 raft_consensus.cc:482] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:23.250705 30529 raft_consensus.cc:504] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.256225 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.247780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:39733 (local address 127.27.84.194:45303)
0808 15:48:23.247992 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.248000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.248656 (+ 656us) server_negotiation.cc:407] Connection header received
0808 15:48:23.248803 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.248830 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.248945 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.249044 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.249564 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.250655 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.253095 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.254524 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.254786 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.254890 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.255654 (+ 764us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.255850 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 15:48:23.256768 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.247526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43277 (local address 127.27.84.195:48075)
0808 15:48:23.247953 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.247975 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.248129 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.248472 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.248478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.248521 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.248767 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.248781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.250263 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.250276 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.252471 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.252493 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.255720 (+ 3227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.255728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.256008 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.256056 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.256164 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.256377 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253}
I20260808 15:48:23.257701 30514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.247685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:48075 (local address 127.27.84.193:43277)
0808 15:48:23.247943 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.247952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.248229 (+ 277us) server_negotiation.cc:407] Connection header received
0808 15:48:23.248300 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.248303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.248362 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.248421 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.248956 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.250083 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.252679 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.254245 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.254421 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.254533 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.257128 (+ 2595us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.257347 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:48:23.258023 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.247608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:45303 (local address 127.27.84.195:39733)
0808 15:48:23.248521 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.248540 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.248665 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.249065 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.249069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.249089 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.249375 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.249382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.250830 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.250835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.252920 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.252936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.255076 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.255085 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.255353 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.255410 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.257084 (+ 1674us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.257294 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":563,"threads_started":1}
I20260808 15:48:23.258781 29973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa80fabbd114e3d87bccc93c6e2a187" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f1d91ddd4742fc97ee743d67dce33c" is_pre_election: true
I20260808 15:48:23.259272 29973 raft_consensus.cc:2454] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 0.
I20260808 15:48:23.260149 30301 leader_election.cc:302] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f1d91ddd4742fc97ee743d67dce33c, b32c676c5e9d41c89f3d3b1a18d053cb; no voters:
I20260808 15:48:23.260795 30530 raft_consensus.cc:2781] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:23.260794 30529 leader_election.cc:288] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303), b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195:42251)
I20260808 15:48:23.261051 30530 raft_consensus.cc:482] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:23.261248 30530 raft_consensus.cc:3037] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Advancing to term 1
W20260808 15:48:23.263901 30052 tablet.cc:2369] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:23.265612 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa80fabbd114e3d87bccc93c6e2a187" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58" is_pre_election: true
I20260808 15:48:23.265790 30530 raft_consensus.cc:504] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.266341 30203 raft_consensus.cc:2454] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 0.
I20260808 15:48:23.267313 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa80fabbd114e3d87bccc93c6e2a187" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58"
I20260808 15:48:23.267558 30203 raft_consensus.cc:3037] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.267731 30530 leader_election.cc:288] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 election: Requested vote from peers 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303), 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277)
I20260808 15:48:23.268069 29973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa80fabbd114e3d87bccc93c6e2a187" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f1d91ddd4742fc97ee743d67dce33c"
I20260808 15:48:23.268289 29973 raft_consensus.cc:3037] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.271196 30203 raft_consensus.cc:2454] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 1.
I20260808 15:48:23.271661 30299 leader_election.cc:302] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e4e3b29856b409f8baba6bdd369cd58, b32c676c5e9d41c89f3d3b1a18d053cb; no voters:
I20260808 15:48:23.271867 29973 raft_consensus.cc:2454] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 1.
I20260808 15:48:23.272089 30530 raft_consensus.cc:2781] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:23.272893 30530 raft_consensus.cc:686] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 LEADER]: Becoming Leader. State: Replica: b32c676c5e9d41c89f3d3b1a18d053cb, State: Running, Role: LEADER
I20260808 15:48:23.273727 30530 consensus_queue.cc:237] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.277828 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.253598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:45303 (local address 127.27.84.193:59353)
0808 15:48:23.267409 (+ 13811us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.267430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.268152 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.268781 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.268793 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.268830 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.269429 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.269446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.271646 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.271654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.274879 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.274903 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.276824 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.276831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.277023 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.277087 (+ 64us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.277203 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.277385 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13655,"thread_start_us":13477,"threads_started":1}
I20260808 15:48:23.278568 30510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.254789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:59353 (local address 127.27.84.194:45303)
0808 15:48:23.257028 (+ 2239us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.257036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.267515 (+ 10479us) server_negotiation.cc:407] Connection header received
0808 15:48:23.268315 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.268325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.268442 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.268558 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.269658 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.271334 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.275080 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.276371 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.276555 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.276634 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.278109 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.278261 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1930,"thread_start_us":8183,"threads_started":1}
I20260808 15:48:23.279582 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa80fabbd114e3d87bccc93c6e2a187" candidate_uuid: "16f1d91ddd4742fc97ee743d67dce33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58" is_pre_election: true
I20260808 15:48:23.279531 29720 catalog_manager.cc:5495] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb reported cstate change: term changed from 0 to 1, leader changed from <none> to b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195). New cstate: current_term: 1 leader_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } health_report { overall_health: HEALTHY } } }
I20260808 15:48:23.280190 30203 raft_consensus.cc:2379] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16f1d91ddd4742fc97ee743d67dce33c in current term 1: Already voted for candidate b32c676c5e9d41c89f3d3b1a18d053cb in this term.
I20260808 15:48:23.284673 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.255270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:33057 (local address 127.27.84.195:42251)
0808 15:48:23.255556 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.255565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.279220 (+ 23655us) server_negotiation.cc:407] Connection header received
0808 15:48:23.279410 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.279415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.279493 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.279588 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:23.280032 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.281040 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.282817 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.283748 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.283888 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.283938 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.284266 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.284404 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:48:23.284747 30533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.253725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:42251 (local address 127.27.84.193:33057)
0808 15:48:23.279088 (+ 25363us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.279109 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.279254 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.279688 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.279693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.279723 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:23.279895 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.279907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.281196 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.281202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.282651 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.282669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.283925 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.283930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.284074 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.284100 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.284239 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.284404 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25237,"thread_start_us":25703,"threads_started":1}
I20260808 15:48:23.285580 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa80fabbd114e3d87bccc93c6e2a187" candidate_uuid: "16f1d91ddd4742fc97ee743d67dce33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" is_pre_election: true
I20260808 15:48:23.286564 29851 leader_election.cc:302] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16f1d91ddd4742fc97ee743d67dce33c; no voters: 2e4e3b29856b409f8baba6bdd369cd58, b32c676c5e9d41c89f3d3b1a18d053cb
I20260808 15:48:23.287113 30529 raft_consensus.cc:2726] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 15:48:23.329185 30515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.322779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.27.84.195:42251)
0808 15:48:23.322993 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.323001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.323164 (+ 163us) server_negotiation.cc:407] Connection header received
0808 15:48:23.323542 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.323547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.323638 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.323793 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:23.324205 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.324983 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.326420 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.326921 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.327064 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.327700 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:23.328358 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:23.328479 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.328695 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.328777 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:48:23.338021 30203 raft_consensus.cc:1261] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Refusing update from remote peer b32c676c5e9d41c89f3d3b1a18d053cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:23.338202 29973 raft_consensus.cc:1261] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Refusing update from remote peer b32c676c5e9d41c89f3d3b1a18d053cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:23.339028 30530 consensus_queue.cc:1035] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:23.339694 30537 consensus_queue.cc:1035] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:23.371919 29719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:23.372344 29719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:23.376577 29719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: Sending DeleteTablet for 3 replicas of tablet eaa80fabbd114e3d87bccc93c6e2a187
I20260808 15:48:23.378140 30167 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa80fabbd114e3d87bccc93c6e2a187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:23 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49232
I20260808 15:48:23.378159 29935 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa80fabbd114e3d87bccc93c6e2a187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:23 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:60288
I20260808 15:48:23.378542 30393 tablet_service.cc:1514] Processing DeleteTablet for tablet eaa80fabbd114e3d87bccc93c6e2a187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:23 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:39544
I20260808 15:48:23.382691 29719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:23.383659 29719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:23.389987 30554 tablet_replica.cc:330] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: stopping tablet replica
I20260808 15:48:23.390815 30554 raft_consensus.cc:2229] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:23.391757 30554 raft_consensus.cc:2258] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:23.392495 30553 tablet_replica.cc:330] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: stopping tablet replica
I20260808 15:48:23.393052 30553 raft_consensus.cc:2229] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:23.394235 30553 raft_consensus.cc:2258] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:23.396926 29934 tablet_service.cc:1467] Processing CreateTablet for tablet 896f62e8482648e0a4714632273ff44f (DEFAULT_TABLE table=test-table [id=8d25ec9c99a64a96aedc25b8db4e0ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:23.397697 30393 tablet_service.cc:1467] Processing CreateTablet for tablet 896f62e8482648e0a4714632273ff44f (DEFAULT_TABLE table=test-table [id=8d25ec9c99a64a96aedc25b8db4e0ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:23.397799 29934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896f62e8482648e0a4714632273ff44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.398330 30393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896f62e8482648e0a4714632273ff44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.399117 30555 tablet_replica.cc:330] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: stopping tablet replica
I20260808 15:48:23.396760 30164 tablet_service.cc:1467] Processing CreateTablet for tablet 896f62e8482648e0a4714632273ff44f (DEFAULT_TABLE table=test-table [id=8d25ec9c99a64a96aedc25b8db4e0ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:23.401528 30164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 896f62e8482648e0a4714632273ff44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.404064 30554 ts_tablet_manager.cc:1905] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:23.399673 30555 raft_consensus.cc:2229] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:23.406113 30555 raft_consensus.cc:2258] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:23.406738 30520 tablet_bootstrap.cc:492] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Bootstrap starting.
I20260808 15:48:23.408525 30520 tablet_bootstrap.cc:654] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.411434 30516 tablet_bootstrap.cc:492] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Bootstrap starting.
I20260808 15:48:23.413345 30516 tablet_bootstrap.cc:654] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.414448 30520 tablet_bootstrap.cc:492] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: No bootstrap required, opened a new log
I20260808 15:48:23.414624 30520 ts_tablet_manager.cc:1397] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260808 15:48:23.415486 30520 raft_consensus.cc:348] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.415738 30520 raft_consensus.cc:374] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.415850 30520 raft_consensus.cc:729] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f1d91ddd4742fc97ee743d67dce33c, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.416146 30520 consensus_queue.cc:260] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.416570 30553 ts_tablet_manager.cc:1905] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:23.419234 30520 ts_tablet_manager.cc:1428] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260808 15:48:23.422757 30555 ts_tablet_manager.cc:1905] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:23.427735 30521 tablet_bootstrap.cc:492] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Bootstrap starting.
I20260808 15:48:23.429733 30521 tablet_bootstrap.cc:654] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.430699 30555 ts_tablet_manager.cc:1918] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:23.430902 30555 log.cc:1196] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/eaa80fabbd114e3d87bccc93c6e2a187
I20260808 15:48:23.431466 30555 ts_tablet_manager.cc:1939] T eaa80fabbd114e3d87bccc93c6e2a187 P 16f1d91ddd4742fc97ee743d67dce33c: Deleting consensus metadata
I20260808 15:48:23.433565 29704 catalog_manager.cc:4841] TS 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277): tablet eaa80fabbd114e3d87bccc93c6e2a187 (table test-table [id=057c352293444d109894c65d361fd42d]) successfully deleted
I20260808 15:48:23.434839 30553 ts_tablet_manager.cc:1918] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:23.435086 30553 log.cc:1196] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/eaa80fabbd114e3d87bccc93c6e2a187
I20260808 15:48:23.435326 30516 tablet_bootstrap.cc:492] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: No bootstrap required, opened a new log
I20260808 15:48:23.435532 30516 ts_tablet_manager.cc:1397] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.008s
I20260808 15:48:23.435758 30553 ts_tablet_manager.cc:1939] T eaa80fabbd114e3d87bccc93c6e2a187 P 2e4e3b29856b409f8baba6bdd369cd58: Deleting consensus metadata
I20260808 15:48:23.436322 30516 raft_consensus.cc:348] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.436543 30516 raft_consensus.cc:374] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.436650 30516 raft_consensus.cc:729] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4e3b29856b409f8baba6bdd369cd58, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.436942 30516 consensus_queue.cc:260] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.437475 30516 ts_tablet_manager.cc:1428] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:48:23.437911 30554 ts_tablet_manager.cc:1918] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:23.438297 30554 log.cc:1196] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/eaa80fabbd114e3d87bccc93c6e2a187
I20260808 15:48:23.438983 30554 ts_tablet_manager.cc:1939] T eaa80fabbd114e3d87bccc93c6e2a187 P b32c676c5e9d41c89f3d3b1a18d053cb: Deleting consensus metadata
I20260808 15:48:23.439038 30521 tablet_bootstrap.cc:492] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: No bootstrap required, opened a new log
I20260808 15:48:23.439474 30521 ts_tablet_manager.cc:1397] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260808 15:48:23.440352 30521 raft_consensus.cc:348] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.440598 30521 raft_consensus.cc:374] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.440722 30521 raft_consensus.cc:729] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32c676c5e9d41c89f3d3b1a18d053cb, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.441256 29703 catalog_manager.cc:4841] TS b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195:42251): tablet eaa80fabbd114e3d87bccc93c6e2a187 (table test-table [id=057c352293444d109894c65d361fd42d]) successfully deleted
I20260808 15:48:23.441988 29701 catalog_manager.cc:4841] TS 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303): tablet eaa80fabbd114e3d87bccc93c6e2a187 (table test-table [id=057c352293444d109894c65d361fd42d]) successfully deleted
I20260808 15:48:23.442297 30521 consensus_queue.cc:260] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.442971 30521 ts_tablet_manager.cc:1428] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:23.642148 30537 raft_consensus.cc:482] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:23.642422 30537 raft_consensus.cc:504] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.643265 30537 leader_election.cc:288] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277), 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303)
I20260808 15:48:23.643962 29973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896f62e8482648e0a4714632273ff44f" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f1d91ddd4742fc97ee743d67dce33c" is_pre_election: true
I20260808 15:48:23.644029 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896f62e8482648e0a4714632273ff44f" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58" is_pre_election: true
I20260808 15:48:23.644272 29973 raft_consensus.cc:2454] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 0.
I20260808 15:48:23.644320 30203 raft_consensus.cc:2454] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 0.
I20260808 15:48:23.644852 30299 leader_election.cc:302] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16f1d91ddd4742fc97ee743d67dce33c, 2e4e3b29856b409f8baba6bdd369cd58, b32c676c5e9d41c89f3d3b1a18d053cb; no voters:
I20260808 15:48:23.645243 30537 raft_consensus.cc:2781] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:23.645359 30537 raft_consensus.cc:482] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:23.645498 30537 raft_consensus.cc:3037] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.648131 30537 raft_consensus.cc:504] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.649025 30537 leader_election.cc:288] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 election: Requested vote from peers 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277), 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303)
I20260808 15:48:23.649435 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896f62e8482648e0a4714632273ff44f" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58"
I20260808 15:48:23.649427 29973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "896f62e8482648e0a4714632273ff44f" candidate_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16f1d91ddd4742fc97ee743d67dce33c"
I20260808 15:48:23.649657 30203 raft_consensus.cc:3037] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.649657 29973 raft_consensus.cc:3037] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.652890 30203 raft_consensus.cc:2454] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 1.
I20260808 15:48:23.652884 29973 raft_consensus.cc:2454] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32c676c5e9d41c89f3d3b1a18d053cb in term 1.
I20260808 15:48:23.653342 30301 leader_election.cc:302] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f1d91ddd4742fc97ee743d67dce33c, b32c676c5e9d41c89f3d3b1a18d053cb; no voters:
I20260808 15:48:23.653679 30537 raft_consensus.cc:2781] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:23.653894 30537 raft_consensus.cc:686] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 LEADER]: Becoming Leader. State: Replica: b32c676c5e9d41c89f3d3b1a18d053cb, State: Running, Role: LEADER
I20260808 15:48:23.654201 30537 consensus_queue.cc:237] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.657258 29720 catalog_manager.cc:5495] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb reported cstate change: term changed from 0 to 1, leader changed from <none> to b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195). New cstate: current_term: 1 leader_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } health_report { overall_health: HEALTHY } } }
Aug 08 15:48:23 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204100, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:23.757323 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.741318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.27.84.254:42895)
0808 15:48:23.742363 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.742370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.742396 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:48:23.742437 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.742440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.742492 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.742609 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:23.743950 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.744784 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.746075 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.746509 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.746672 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.750925 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:23.750931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:23.750935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:23.750968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:23.753970 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:23.755050 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:23.755054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:23.755056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:23.755194 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:23.755725 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:23.755729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:23.755731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:23.756063 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:23.756193 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.756777 (+ 584us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.757007 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":726,"threads_started":1}
I20260808 15:48:23.757841 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.741255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42895 (local address 127.0.0.1:47794)
0808 15:48:23.742001 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.742030 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.742224 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.742675 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.742679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.743518 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:23.743749 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.743759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.744927 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.744934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.745926 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.745937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.746715 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.746721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.746902 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.747628 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:23.747645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:23.750654 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:23.754162 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:23.754180 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:23.754189 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:23.754911 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:23.755336 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:23.755343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:23.755347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:23.755560 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:23.756235 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:23.756246 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:23.756484 (+ 238us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.756690 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.757232 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":144,"thread_start_us":401,"threads_started":1}
I20260808 15:48:23.763803 29720 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:47794:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:48:23.764945 29720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:23.772737 29934 tablet_service.cc:1467] Processing CreateTablet for tablet bca7d1b0cf264fefa8f978717b7d4ad5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=125179d76dda4e4bb8ed6983b195a2fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:48:23.773331 29934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca7d1b0cf264fefa8f978717b7d4ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.775180 30393 tablet_service.cc:1467] Processing CreateTablet for tablet bca7d1b0cf264fefa8f978717b7d4ad5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=125179d76dda4e4bb8ed6983b195a2fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:48:23.775846 30393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca7d1b0cf264fefa8f978717b7d4ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.779812 30164 tablet_service.cc:1467] Processing CreateTablet for tablet bca7d1b0cf264fefa8f978717b7d4ad5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=125179d76dda4e4bb8ed6983b195a2fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:48:23.780452 30164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca7d1b0cf264fefa8f978717b7d4ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:23.786070 30520 tablet_bootstrap.cc:492] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: Bootstrap starting.
I20260808 15:48:23.787978 30520 tablet_bootstrap.cc:654] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.790853 30516 tablet_bootstrap.cc:492] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: Bootstrap starting.
I20260808 15:48:23.792193 30521 tablet_bootstrap.cc:492] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: Bootstrap starting.
I20260808 15:48:23.792641 30516 tablet_bootstrap.cc:654] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.794677 30521 tablet_bootstrap.cc:654] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:23.799412 30521 tablet_bootstrap.cc:492] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: No bootstrap required, opened a new log
I20260808 15:48:23.799512 30516 tablet_bootstrap.cc:492] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: No bootstrap required, opened a new log
I20260808 15:48:23.799595 30521 ts_tablet_manager.cc:1397] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260808 15:48:23.799667 30516 ts_tablet_manager.cc:1397] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260808 15:48:23.800418 30521 raft_consensus.cc:348] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.800415 30516 raft_consensus.cc:348] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.800678 30516 raft_consensus.cc:374] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.800681 30521 raft_consensus.cc:374] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.800756 30516 raft_consensus.cc:729] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e4e3b29856b409f8baba6bdd369cd58, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.800774 30521 raft_consensus.cc:729] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32c676c5e9d41c89f3d3b1a18d053cb, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.801021 30516 consensus_queue.cc:260] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.801035 30521 consensus_queue.cc:260] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.801604 30516 ts_tablet_manager.cc:1428] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260808 15:48:23.801640 30521 ts_tablet_manager.cc:1428] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:48:23.801930 30520 tablet_bootstrap.cc:492] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: No bootstrap required, opened a new log
I20260808 15:48:23.802201 30520 ts_tablet_manager.cc:1397] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260808 15:48:23.802055 30528 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.802387 30528 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.802889 30537 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.803205 30537 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.804498 30520 raft_consensus.cc:348] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.804858 30520 raft_consensus.cc:374] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:23.804955 30520 raft_consensus.cc:729] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f1d91ddd4742fc97ee743d67dce33c, State: Initialized, Role: FOLLOWER
I20260808 15:48:23.805217 30520 consensus_queue.cc:260] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.805840 30520 ts_tablet_manager.cc:1428] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260808 15:48:23.805719 30529 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.806028 30529 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.844579 30529 raft_consensus.cc:482] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:23.844811 30529 raft_consensus.cc:504] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.845561 30529 leader_election.cc:288] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303), b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195:42251)
I20260808 15:48:23.846179 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca7d1b0cf264fefa8f978717b7d4ad5" candidate_uuid: "16f1d91ddd4742fc97ee743d67dce33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" is_pre_election: true
I20260808 15:48:23.846150 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca7d1b0cf264fefa8f978717b7d4ad5" candidate_uuid: "16f1d91ddd4742fc97ee743d67dce33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58" is_pre_election: true
I20260808 15:48:23.846465 30203 raft_consensus.cc:2454] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f1d91ddd4742fc97ee743d67dce33c in term 0.
I20260808 15:48:23.846498 30418 raft_consensus.cc:2454] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16f1d91ddd4742fc97ee743d67dce33c in term 0.
I20260808 15:48:23.847045 29851 leader_election.cc:302] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16f1d91ddd4742fc97ee743d67dce33c, 2e4e3b29856b409f8baba6bdd369cd58, b32c676c5e9d41c89f3d3b1a18d053cb; no voters:
I20260808 15:48:23.847359 30529 raft_consensus.cc:2781] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:23.847496 30529 raft_consensus.cc:482] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:23.847605 30529 raft_consensus.cc:3037] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.851609 30529 raft_consensus.cc:504] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.852226 30529 leader_election.cc:288] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [CANDIDATE]: Term 1 election: Requested vote from peers 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303), b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195:42251)
I20260808 15:48:23.852806 30203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca7d1b0cf264fefa8f978717b7d4ad5" candidate_uuid: "16f1d91ddd4742fc97ee743d67dce33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e4e3b29856b409f8baba6bdd369cd58"
I20260808 15:48:23.852834 30418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca7d1b0cf264fefa8f978717b7d4ad5" candidate_uuid: "16f1d91ddd4742fc97ee743d67dce33c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb"
I20260808 15:48:23.853062 30203 raft_consensus.cc:3037] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.853111 30418 raft_consensus.cc:3037] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:23.855311 30418 raft_consensus.cc:2454] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f1d91ddd4742fc97ee743d67dce33c in term 1.
I20260808 15:48:23.855362 30203 raft_consensus.cc:2454] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16f1d91ddd4742fc97ee743d67dce33c in term 1.
I20260808 15:48:23.855855 29851 leader_election.cc:302] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16f1d91ddd4742fc97ee743d67dce33c, b32c676c5e9d41c89f3d3b1a18d053cb; no voters:
I20260808 15:48:23.856185 30529 raft_consensus.cc:2781] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:23.856664 30529 raft_consensus.cc:686] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [term 1 LEADER]: Becoming Leader. State: Replica: 16f1d91ddd4742fc97ee743d67dce33c, State: Running, Role: LEADER
I20260808 15:48:23.857097 30529 consensus_queue.cc:237] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } }
I20260808 15:48:23.859268 30576 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: TxnStatusTablet state changed. Reason: New leader 16f1d91ddd4742fc97ee743d67dce33c. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.859570 30576 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:48:23.860236 30578 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:48:23.860569 30578 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:48:23.861217 29719 catalog_manager.cc:5495] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c reported cstate change: term changed from 0 to 1, leader changed from <none> to 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193). New cstate: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } health_report { overall_health: UNKNOWN } } }
I20260808 15:48:23.898646 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.888133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43277 (local address 127.0.0.1:60296)
0808 15:48:23.888364 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.888390 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.888508 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.889980 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.889987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.890005 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:23.890237 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.890252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.891978 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.891986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.893843 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.893864 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.895889 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.895895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.896137 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.896236 (+ 99us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:23.897872 (+ 1636us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:23.897905 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.897980 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.898202 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 15:48:23.899513 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.888245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60296 (local address 127.27.84.193:43277)
0808 15:48:23.889486 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.889495 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.889516 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:48:23.889638 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.889643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.889746 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.889936 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:23.890464 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.891799 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.894027 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.895641 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.895888 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.896568 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:23.897685 (+ 1117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:23.898939 (+ 1254us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.898993 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.899088 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1092,"thread_start_us":899,"threads_started":1}
I20260808 15:48:23.906492 30203 raft_consensus.cc:1261] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Refusing update from remote peer 16f1d91ddd4742fc97ee743d67dce33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:23.906563 30418 raft_consensus.cc:1261] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Refusing update from remote peer 16f1d91ddd4742fc97ee743d67dce33c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:23.907464 30529 consensus_queue.cc:1035] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:23.907904 30576 consensus_queue.cc:1035] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:23.915565 30537 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: TxnStatusTablet state changed. Reason: New leader 16f1d91ddd4742fc97ee743d67dce33c. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.915904 30537 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.931223 30576 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.931654 30576 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:48:23.941073 30528 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: TxnStatusTablet state changed. Reason: New leader 16f1d91ddd4742fc97ee743d67dce33c. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.941401 30528 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.950263 30537 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.950584 30537 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P b32c676c5e9d41c89f3d3b1a18d053cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.960443 30576 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.960767 30576 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 16f1d91ddd4742fc97ee743d67dce33c: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:48:23.961227 30588 mvcc.cc:205] Tried to move back new op lower bound from 7316292009591521280 to 7316292009402302464. Current Snapshot: MvccSnapshot[applied={T|T < 7316292009591521280}]
I20260808 15:48:23.962481 30528 tablet_replica.cc:439] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16f1d91ddd4742fc97ee743d67dce33c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 } } peers { permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 } } peers { permanent_uuid: "b32c676c5e9d41c89f3d3b1a18d053cb" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42251 } } }
I20260808 15:48:23.962747 30528 tablet_replica.cc:442] T bca7d1b0cf264fefa8f978717b7d4ad5 P 2e4e3b29856b409f8baba6bdd369cd58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:23.980543 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.972929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:50333 (local address 127.27.84.193:43277)
0808 15:48:23.973173 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:23.973182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:23.975309 (+ 2127us) server_negotiation.cc:407] Connection header received
0808 15:48:23.975541 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:23.975546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:23.975615 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:23.975719 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:23.976426 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.977223 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.978638 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.979094 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.979244 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.979739 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:23.979849 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:23.979968 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:23.980101 (+ 133us) server_negotiation.cc:299] Negotiation successful
0808 15:48:23.980185 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:48:23.980633 30601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:23.972812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43277 (local address 127.27.84.195:50333)
0808 15:48:23.975185 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:23.975210 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:23.975384 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:23.975751 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:23.975756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:23.975771 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:23.976257 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.976275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.977392 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.977398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.978460 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:23.978473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:23.979292 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:23.979300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:23.979463 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:23.979524 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:23.979964 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:23.979979 (+ 15us) client_negotiation.cc:769] Sending connection context
0808 15:48:23.980078 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 15:48:23.980224 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2193,"thread_start_us":2025,"threads_started":1}
I20260808 15:48:23.990847 29973 raft_consensus.cc:1261] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Refusing update from remote peer b32c676c5e9d41c89f3d3b1a18d053cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:23.990973 30203 raft_consensus.cc:1261] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Refusing update from remote peer b32c676c5e9d41c89f3d3b1a18d053cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:23.991600 30530 consensus_queue.cc:1035] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "16f1d91ddd4742fc97ee743d67dce33c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:23.991917 30537 consensus_queue.cc:1035] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e4e3b29856b409f8baba6bdd369cd58" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:23.999099 30589 mvcc.cc:205] Tried to move back new op lower bound from 7316292009941028864 to 7316292008570081280. Current Snapshot: MvccSnapshot[applied={T|T < 7316292009941028864}]
I20260808 15:48:24.003307 30586 mvcc.cc:205] Tried to move back new op lower bound from 7316292009941028864 to 7316292008570081280. Current Snapshot: MvccSnapshot[applied={T|T < 7316292009941028864}]
I20260808 15:48:24.035038 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.026062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:42963 (local address 127.27.84.195:42251)
0808 15:48:24.027558 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:24.027564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:24.028417 (+ 853us) server_negotiation.cc:407] Connection header received
0808 15:48:24.028491 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:24.028496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:24.028580 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:24.028726 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:24.029540 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.030882 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.032773 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.033471 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.033615 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.034155 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:24.034307 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:24.034421 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:24.034573 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 15:48:24.034697 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1362,"thread_start_us":1225,"threads_started":1}
I20260808 15:48:24.035197 30611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.026213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:42251 (local address 127.27.84.193:42963)
0808 15:48:24.028105 (+ 1892us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:24.028126 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:24.028282 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:24.028751 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:24.028757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:24.028772 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:24.029296 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:24.029309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.031054 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.031060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:24.032595 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:24.032609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.033692 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.033699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:24.033844 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.033911 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:24.034428 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:24.034442 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 15:48:24.034536 (+ 94us) client_negotiation.cc:240] Negotiation successful
0808 15:48:24.034737 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1673,"thread_start_us":1507,"threads_started":1}
I20260808 15:48:24.076351 29719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:24.076604 29719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:24.080166 29719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f7a2baefdb44fb4bc8468a0d1ca66f0: Sending DeleteTablet for 3 replicas of tablet 896f62e8482648e0a4714632273ff44f
I20260808 15:48:24.080900 29934 tablet_service.cc:1514] Processing DeleteTablet for tablet 896f62e8482648e0a4714632273ff44f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:24 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:60288
I20260808 15:48:24.081189 30393 tablet_service.cc:1514] Processing DeleteTablet for tablet 896f62e8482648e0a4714632273ff44f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:24 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:39544
I20260808 15:48:24.081210 30164 tablet_service.cc:1514] Processing DeleteTablet for tablet 896f62e8482648e0a4714632273ff44f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:24 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49232
I20260808 15:48:24.099292 30613 tablet_replica.cc:330] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: stopping tablet replica
I20260808 15:48:24.099577 30613 raft_consensus.cc:2229] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:24.100024 30613 raft_consensus.cc:2258] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:24.101689 30615 tablet_replica.cc:330] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: stopping tablet replica
I20260808 15:48:24.101984 30615 raft_consensus.cc:2229] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:24.102306 30615 raft_consensus.cc:2258] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:24.102986 30616 tablet_replica.cc:330] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: stopping tablet replica
I20260808 15:48:24.103206 30616 raft_consensus.cc:2229] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:24.103657 30616 raft_consensus.cc:2258] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:24.103791 30615 ts_tablet_manager.cc:1905] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:24.105000 30613 ts_tablet_manager.cc:1905] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:24.110004 30616 ts_tablet_manager.cc:1905] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:24.111678 30615 ts_tablet_manager.cc:1918] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:24.111932 30615 log.cc:1196] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/896f62e8482648e0a4714632273ff44f
I20260808 15:48:24.112517 30615 ts_tablet_manager.cc:1939] T 896f62e8482648e0a4714632273ff44f P 2e4e3b29856b409f8baba6bdd369cd58: Deleting consensus metadata
I20260808 15:48:24.114081 29701 catalog_manager.cc:4841] TS 2e4e3b29856b409f8baba6bdd369cd58 (127.27.84.194:45303): tablet 896f62e8482648e0a4714632273ff44f (table test-table [id=8d25ec9c99a64a96aedc25b8db4e0ffd]) successfully deleted
Aug 08 15:48:24 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:24.128399 30613 ts_tablet_manager.cc:1918] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:24.128567 30613 log.cc:1196] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/896f62e8482648e0a4714632273ff44f
I20260808 15:48:24.129086 30613 ts_tablet_manager.cc:1939] T 896f62e8482648e0a4714632273ff44f P 16f1d91ddd4742fc97ee743d67dce33c: Deleting consensus metadata
I20260808 15:48:24.130470 29704 catalog_manager.cc:4841] TS 16f1d91ddd4742fc97ee743d67dce33c (127.27.84.193:43277): tablet 896f62e8482648e0a4714632273ff44f (table test-table [id=8d25ec9c99a64a96aedc25b8db4e0ffd]) successfully deleted
I20260808 15:48:24.131542 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.100317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.27.84.193:43277)
0808 15:48:24.100584 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:24.100593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:24.108105 (+ 7512us) server_negotiation.cc:407] Connection header received
0808 15:48:24.108613 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:24.108619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:24.108738 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:24.108820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:24.111080 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.112301 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.114913 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.116200 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.116454 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.121978 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:24.122000 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:24.122024 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:24.122079 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:24.126658 (+ 4579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.127675 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.127686 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.127690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.127889 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.128427 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.128433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.128436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.129134 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:24.129545 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:24.130870 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0808 15:48:24.131121 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260808 15:48:24.133294 30045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60306
I20260808 15:48:24.135212 30616 ts_tablet_manager.cc:1918] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:24.135381 30616 log.cc:1196] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/896f62e8482648e0a4714632273ff44f
I20260808 15:48:24.135898 30616 ts_tablet_manager.cc:1939] T 896f62e8482648e0a4714632273ff44f P b32c676c5e9d41c89f3d3b1a18d053cb: Deleting consensus metadata
I20260808 15:48:24.137181 29703 catalog_manager.cc:4841] TS b32c676c5e9d41c89f3d3b1a18d053cb (127.27.84.195:42251): tablet 896f62e8482648e0a4714632273ff44f (table test-table [id=8d25ec9c99a64a96aedc25b8db4e0ffd]) successfully deleted
I20260808 15:48:24.150884 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.137516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.27.84.254:42895)
0808 15:48:24.137737 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:24.137762 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:24.138144 (+ 382us) server_negotiation.cc:407] Connection header received
0808 15:48:24.138364 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:24.138370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:24.138458 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:24.138564 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:24.139905 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.140653 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.141905 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.142366 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.142505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.145007 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:24.145014 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:24.145019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:24.145062 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:24.147901 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.148624 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.148632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.148636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.148805 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.149281 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.149287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.149291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.149611 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:24.149886 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:24.150247 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 15:48:24.150499 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260808 15:48:24.151489 29719 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47796
I20260808 15:48:24.154726 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29827
I20260808 15:48:24.169955 30585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.155670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.27.84.193:43277)
0808 15:48:24.155880 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:24.155892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:24.156196 (+ 304us) server_negotiation.cc:407] Connection header received
0808 15:48:24.156349 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:24.156360 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:24.156446 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:24.156561 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:24.157754 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.158892 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.160114 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.160851 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.160978 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.163402 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:24.163410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:24.163416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:24.163457 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:24.167058 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.167787 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.167793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.167797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.167955 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.168394 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.168399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.168402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.168763 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:24.169046 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:24.169347 (+ 301us) server_negotiation.cc:299] Negotiation successful
0808 15:48:24.169603 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 15:48:24.170680 30045 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60308
W20260808 15:48:24.171304 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29827: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:24.171546 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29827
I20260808 15:48:24.203135 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30055
Aug 08 15:48:24 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:24.220508 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.204159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.27.84.194:45303)
0808 15:48:24.204724 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:24.204728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:24.204746 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:24.204805 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:24.204808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:24.204873 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:24.204969 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:24.206210 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.207146 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.208489 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.208946 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.209106 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.213621 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:24.213639 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:24.213657 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:24.213703 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:24.217206 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.218119 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.218127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.218130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.218286 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.218994 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.218998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.219000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.219438 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:24.219740 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:24.220017 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 15:48:24.220212 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":303,"threads_started":1}
W20260808 15:48:24.221208 30268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49238
W20260808 15:48:24.221876 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30055: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:24.222036 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30055
I20260808 15:48:24.252317 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30279
Aug 08 15:48:24 dist-test-slave-twwt krb5kdc[29653](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204102, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:24.267374 30610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:24.253155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39566 (local address 127.27.84.195:42251)
0808 15:48:24.253345 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:24.253349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:24.253367 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:24.253447 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:24.253451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:24.253514 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:24.253591 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:24.255005 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.255716 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.257135 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:24.257548 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:24.257641 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:24.261873 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:24.261890 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:24.261907 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:24.261944 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:24.264937 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.265538 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.265545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.265548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.265685 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:24.266027 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:24.266030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:24.266032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:24.266382 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:24.266601 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:24.266936 (+ 335us) server_negotiation.cc:299] Negotiation successful
0808 15:48:24.267110 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:48:24.268121 30478 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39566
W20260808 15:48:24.268850 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30279: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:24.268986 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30279
I20260808 15:48:24.302544 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29674
I20260808 15:48:24.303860 29783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:24.423858 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29674
2026-08-08T15:48:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5405 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:24 dist-test-slave-twwt krb5kdc[30629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:24 dist-test-slave-twwt krb5kdc[30629](info): set up 2 sockets
Aug 08 15:48:24 dist-test-slave-twwt krb5kdc[30629](info): commencing operation
krb5kdc: starting...
W20260808 15:48:25.779920 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:25 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:26.023451 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:33643
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:33643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:26.361304 30652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:26.361668 30652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:26.361847 30652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:26.369038 30652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:26.369194 30652 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:48:26.369287 30652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:26.369405 30652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:26.369494 30652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:26.369583 30652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:26.380555 30652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:33643
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:33643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:26.381960 30652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:26.383397 30652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:26.396574 30664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:26.396616 30666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:26.396664 30669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:26.397584 30652 server_base.cc:1034] running on GCE node
I20260808 15:48:26.398463 30652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:26.400038 30652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:26.401309 30652 hybrid_clock.cc:648] HybridClock initialized: now 1786204106401230 us; error 47 us; skew 500 ppm
I20260808 15:48:26.401684 30652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:26 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:26.405105 30652 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:26.406845 30652 webserver.cc:460] Webserver started at http://127.27.84.254:42675/ using document root <none> and password file <none>
I20260808 15:48:26.407610 30652 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:26.407752 30652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:26.408092 30652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:26.411276 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "9056937fb38d4556870efb71ddb73b8a"
format_stamp: "Formatted at 2026-08-08 15:48:26 on dist-test-slave-twwt"
server_key: "d76651a0c278e80d58e1806e2d809924"
server_key_iv: "f190a7a9e886e044571fb7154a44cf25"
server_key_version: "encryptionkey@0"
I20260808 15:48:26.412087 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "9056937fb38d4556870efb71ddb73b8a"
format_stamp: "Formatted at 2026-08-08 15:48:26 on dist-test-slave-twwt"
server_key: "d76651a0c278e80d58e1806e2d809924"
server_key_iv: "f190a7a9e886e044571fb7154a44cf25"
server_key_version: "encryptionkey@0"
I20260808 15:48:26.417447 30652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:48:26.421082 30679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:26.422453 30652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:26.422706 30652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "9056937fb38d4556870efb71ddb73b8a"
format_stamp: "Formatted at 2026-08-08 15:48:26 on dist-test-slave-twwt"
server_key: "d76651a0c278e80d58e1806e2d809924"
server_key_iv: "f190a7a9e886e044571fb7154a44cf25"
server_key_version: "encryptionkey@0"
I20260808 15:48:26.422976 30652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:26.439491 30652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:26.444207 30652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:26.444536 30652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:26.466943 30652 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:33643
I20260808 15:48:26.466995 30779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:33643 every 8 connection(s)
I20260808 15:48:26.468395 30652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:26.472546 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:26.477226 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30652
I20260808 15:48:26.477466 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:26.478314 27987 external_mini_cluster.cc:1428] Setting key fd4c7b8ae852c22772cbaa4407aab30e
I20260808 15:48:26.485035 30781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Bootstrap starting.
I20260808 15:48:26.488462 30781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:26.489629 30781 log.cc:824] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:26.493232 30781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: No bootstrap required, opened a new log
Aug 08 15:48:26 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:26.502842 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:26.480681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.27.84.254:33643)
0808 15:48:26.481523 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:26.481539 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:26.481585 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:48:26.483609 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:26.483634 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:26.484053 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:26.484459 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:26.486062 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:26.487505 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:26.488912 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:26.489632 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:26.489856 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:26.494509 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:26.494517 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:26.494534 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:26.494570 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:26.497930 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:26.499001 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:26.499009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:26.499016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:26.499212 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:26.499706 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:26.499709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:26.499712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:26.500339 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:26.500570 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:26.501122 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 15:48:26.501539 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":364,"threads_started":1}
I20260808 15:48:26.503768 30781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056937fb38d4556870efb71ddb73b8a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33643 } }
I20260808 15:48:26.504185 30781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:26.504350 30781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056937fb38d4556870efb71ddb73b8a, State: Initialized, Role: FOLLOWER
I20260808 15:48:26.505002 30781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056937fb38d4556870efb71ddb73b8a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33643 } }
I20260808 15:48:26.505331 30781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:26.505501 30781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:26.505721 30781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:26.508306 30781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056937fb38d4556870efb71ddb73b8a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33643 } }
I20260808 15:48:26.508864 30781 leader_election.cc:302] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9056937fb38d4556870efb71ddb73b8a; no voters:
I20260808 15:48:26.509498 30781 leader_election.cc:288] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:26.509660 30793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:26.510914 30793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [term 1 LEADER]: Becoming Leader. State: Replica: 9056937fb38d4556870efb71ddb73b8a, State: Running, Role: LEADER
I20260808 15:48:26.511483 30793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056937fb38d4556870efb71ddb73b8a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33643 } }
I20260808 15:48:26.512001 30781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:26.514362 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9056937fb38d4556870efb71ddb73b8a. Latest consensus state: current_term: 1 leader_uuid: "9056937fb38d4556870efb71ddb73b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056937fb38d4556870efb71ddb73b8a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33643 } } }
I20260808 15:48:26.514809 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:26.514348 30794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9056937fb38d4556870efb71ddb73b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9056937fb38d4556870efb71ddb73b8a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33643 } } }
I20260808 15:48:26.515270 30794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:26.522521 30803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:26.528337 30803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:26.537930 30803 catalog_manager.cc:1347] Generated new cluster ID: 30cd5de4bba54b95b913f80098340601
I20260808 15:48:26.538082 30803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:26.560688 30803 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:26.562150 30803 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:26.568398 30803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Generated new TSK 0
I20260808 15:48:26.569537 30803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:26.635792 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33643
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:26.990172 30828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:26.990475 30828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:26.990581 30828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:26.997983 30828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:26.998106 30828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:26.998195 30828 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:48:26.998251 30828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:48:26.998315 30828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:27.009605 30828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:33643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:27.011174 30828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:27.012723 30828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:27.026417 30841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:27.026443 30836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:27.026535 30835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:27.028177 30828 server_base.cc:1034] running on GCE node
I20260808 15:48:27.028870 30828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:27.030014 30828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:27.031234 30828 hybrid_clock.cc:648] HybridClock initialized: now 1786204107031181 us; error 46 us; skew 500 ppm
I20260808 15:48:27.031618 30828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:27 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:27.042668 30828 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:27.044387 30828 webserver.cc:460] Webserver started at http://127.27.84.193:44441/ using document root <none> and password file <none>
I20260808 15:48:27.045140 30828 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:27.045302 30828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:27.045655 30828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:27.048821 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "4c3a7d68ace9422fb626b4e15ebdc9df"
format_stamp: "Formatted at 2026-08-08 15:48:27 on dist-test-slave-twwt"
server_key: "b082f589675e152bb7f4ee4c6a5a9a9a"
server_key_iv: "0359842af054e1a614885d7d5b6731bb"
server_key_version: "encryptionkey@0"
I20260808 15:48:27.049623 30828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "4c3a7d68ace9422fb626b4e15ebdc9df"
format_stamp: "Formatted at 2026-08-08 15:48:27 on dist-test-slave-twwt"
server_key: "b082f589675e152bb7f4ee4c6a5a9a9a"
server_key_iv: "0359842af054e1a614885d7d5b6731bb"
server_key_version: "encryptionkey@0"
I20260808 15:48:27.055156 30828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:48:27.058808 30856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:27.060290 30828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:27.060482 30828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "4c3a7d68ace9422fb626b4e15ebdc9df"
format_stamp: "Formatted at 2026-08-08 15:48:27 on dist-test-slave-twwt"
server_key: "b082f589675e152bb7f4ee4c6a5a9a9a"
server_key_iv: "0359842af054e1a614885d7d5b6731bb"
server_key_version: "encryptionkey@0"
I20260808 15:48:27.060704 30828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:27.092918 30828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:27.097612 30828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:27.097934 30828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:27.100915 30828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:27.101073 30828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:27.101224 30828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:27.101317 30828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:48:27 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:27.129305 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.101984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:54421 (local address 127.27.84.254:33643)
0808 15:48:27.102506 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:27.102511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:27.103757 (+ 1246us) server_negotiation.cc:407] Connection header received
0808 15:48:27.106114 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:27.106119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:27.106219 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:27.106354 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:27.109339 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.110434 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.114052 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.114700 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.114866 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.120416 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:27.120428 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:27.120431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:27.120503 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:27.124054 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.125438 (+ 1384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.125446 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.125451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.125622 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.126313 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.126318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.126322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.126802 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:27.126959 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:27.128365 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0808 15:48:27.128734 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":291,"threads_started":1}
I20260808 15:48:27.130067 30868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.102747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.27.84.193:54421)
0808 15:48:27.103584 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:27.103631 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:27.105796 (+ 2165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:27.106829 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:27.106845 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:27.107891 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:27.109069 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.109104 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.110675 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.110684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.113845 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.113868 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.115008 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.115018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.115418 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.116342 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:27.116393 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:27.120134 (+ 3741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:27.124246 (+ 4112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.124261 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.124290 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.125237 (+ 947us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.125754 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.125759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.125763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.126178 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.127049 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:27.127055 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:27.127784 (+ 729us) client_negotiation.cc:769] Sending connection context
0808 15:48:27.128223 (+ 439us) client_negotiation.cc:240] Negotiation successful
0808 15:48:27.128737 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":384,"threads_started":1}
I20260808 15:48:27.151722 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.136678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:38267 (local address 127.27.84.254:33643)
0808 15:48:27.136940 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:27.136950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:27.137539 (+ 589us) server_negotiation.cc:407] Connection header received
0808 15:48:27.137741 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:27.137744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:27.137819 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:27.137903 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:27.139644 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.140567 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.142138 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.142559 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.142737 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.145443 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:27.145448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:27.145451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:27.145482 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:27.148625 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.149368 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.149375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.149379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.149520 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.150046 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.150051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.150054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.150392 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:27.150571 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:27.151285 (+ 714us) server_negotiation.cc:299] Negotiation successful
0808 15:48:27.151469 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 15:48:27.152439 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.136636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.27.84.193:38267)
0808 15:48:27.137404 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:27.137431 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:27.137615 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:27.137977 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:27.137985 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:27.138828 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:27.139449 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.139472 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.140787 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.140795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.141989 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.142004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.142806 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.142814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.143041 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.143875 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:27.143897 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:27.145239 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:27.148766 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.148772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.148777 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.149233 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.149646 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.149651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.149655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.149864 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.150545 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:27.150556 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:27.150902 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 15:48:27.151150 (+ 248us) client_negotiation.cc:240] Negotiation successful
0808 15:48:27.151626 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":386,"threads_started":1}
I20260808 15:48:27.164201 30828 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:46045
I20260808 15:48:27.164248 31037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:46045 every 8 connection(s)
I20260808 15:48:27.165675 30828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:27.167902 31039 heartbeater.cc:347] Connected to a master server at 127.27.84.254:33643
I20260808 15:48:27.168119 31039 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:27.168718 31039 heartbeater.cc:511] Master 127.27.84.254:33643 requested a full tablet report, sending...
I20260808 15:48:27.170857 30706 ts_manager.cc:194] Registered new tserver with Master: 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045)
I20260808 15:48:27.172384 30706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:54421
I20260808 15:48:27.173578 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30828
I20260808 15:48:27.173784 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:27.174675 27987 external_mini_cluster.cc:1428] Setting key 9aa8dfa34d743f019ddec4664070b0b0
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:27.227109 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33643
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:27.574496 31044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:27.574923 31044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:27.575109 31044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:27.582741 31044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:27.582921 31044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:27.583079 31044 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:48:27.583170 31044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:48:27.583316 31044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:27.594208 31044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:33643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:27.595731 31044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:27.597249 31044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:27.611519 31052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:27.611662 31055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:27.611706 31053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:27.612735 31044 server_base.cc:1034] running on GCE node
I20260808 15:48:27.613492 31044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:27.614583 31044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:27.615939 31044 hybrid_clock.cc:648] HybridClock initialized: now 1786204107615881 us; error 47 us; skew 500 ppm
I20260808 15:48:27.616338 31044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:27 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:27.619773 31044 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:27.621345 31044 webserver.cc:460] Webserver started at http://127.27.84.194:46865/ using document root <none> and password file <none>
I20260808 15:48:27.622067 31044 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:27.622176 31044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:27.622457 31044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:27.625669 31044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "78ca76d31d8e40b89806a9cb624bb630"
format_stamp: "Formatted at 2026-08-08 15:48:27 on dist-test-slave-twwt"
server_key: "fc27228acf363d71ff23c566f051575c"
server_key_iv: "9288a3ee0575e6a5672a33799ac90b46"
server_key_version: "encryptionkey@0"
I20260808 15:48:27.626657 31044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "78ca76d31d8e40b89806a9cb624bb630"
format_stamp: "Formatted at 2026-08-08 15:48:27 on dist-test-slave-twwt"
server_key: "fc27228acf363d71ff23c566f051575c"
server_key_iv: "9288a3ee0575e6a5672a33799ac90b46"
server_key_version: "encryptionkey@0"
I20260808 15:48:27.632606 31044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:48:27.636588 31064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:27.638167 31044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:48:27.638377 31044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "78ca76d31d8e40b89806a9cb624bb630"
format_stamp: "Formatted at 2026-08-08 15:48:27 on dist-test-slave-twwt"
server_key: "fc27228acf363d71ff23c566f051575c"
server_key_iv: "9288a3ee0575e6a5672a33799ac90b46"
server_key_version: "encryptionkey@0"
I20260808 15:48:27.638589 31044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:27.678710 31044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:27.683065 31044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:27.683398 31044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:27.686455 31044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:27.686576 31044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:27.686726 31044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:27.686846 31044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:48:27 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:27.711936 31080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.687049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:59747 (local address 127.27.84.254:33643)
0808 15:48:27.687916 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:27.687921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:27.688807 (+ 886us) server_negotiation.cc:407] Connection header received
0808 15:48:27.691094 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:27.691100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:27.691192 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:27.691328 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:27.693617 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.694728 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.697974 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.698497 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.698705 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.704321 (+ 5616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:27.704328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:27.704333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:27.704381 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:27.708094 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.709222 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.709229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.709236 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.709385 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.709859 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.709864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.709868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.710161 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:27.710337 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:27.711258 (+ 921us) server_negotiation.cc:299] Negotiation successful
0808 15:48:27.711513 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":645,"threads_started":1}
I20260808 15:48:27.713469 31081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.687787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.27.84.194:59747)
0808 15:48:27.688625 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:27.688673 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:27.690751 (+ 2078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:27.691652 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:27.691667 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:27.692487 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:27.693400 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.693438 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.694928 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.694938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.697754 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.697773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.698828 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.698840 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.699318 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.700198 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:27.700239 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:27.704038 (+ 3799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:27.708278 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.708285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.708301 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.709072 (+ 771us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.709512 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.709517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.709525 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.709728 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.710310 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:27.710319 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:27.710765 (+ 446us) client_negotiation.cc:769] Sending connection context
0808 15:48:27.711147 (+ 382us) client_negotiation.cc:240] Negotiation successful
0808 15:48:27.711779 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":376,"threads_started":1}
I20260808 15:48:27.737546 31080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.719743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:58117 (local address 127.27.84.254:33643)
0808 15:48:27.719925 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:27.719933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:27.721023 (+ 1090us) server_negotiation.cc:407] Connection header received
0808 15:48:27.721343 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:27.721348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:27.721413 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:27.721521 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:27.722485 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.723493 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.725538 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.726487 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.726701 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.729458 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:27.729467 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:27.729475 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:27.729525 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:27.733571 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.734544 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.734550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.734556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.734762 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:27.735375 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:27.735381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:27.735383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:27.736136 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:27.736338 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:27.737118 (+ 780us) server_negotiation.cc:299] Negotiation successful
0808 15:48:27.737309 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:48:27.738209 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:27.720252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.27.84.194:58117)
0808 15:48:27.720912 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:27.720933 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:27.721152 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:27.721549 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:27.721554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:27.722047 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:27.722336 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.722346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.723654 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.723660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.725366 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:27.725385 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:27.726760 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:27.726770 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:27.727108 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:27.727881 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:27.727915 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:27.729216 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:27.733736 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.733744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.733750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.734378 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.734925 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:27.734933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:27.734940 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:27.735158 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:27.736434 (+ 1276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:27.736443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:27.736747 (+ 304us) client_negotiation.cc:769] Sending connection context
0808 15:48:27.736974 (+ 227us) client_negotiation.cc:240] Negotiation successful
0808 15:48:27.737472 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":376,"threads_started":1}
I20260808 15:48:27.744824 31044 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:37799
I20260808 15:48:27.744916 31249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:37799 every 8 connection(s)
I20260808 15:48:27.746143 31044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:27.748235 31250 heartbeater.cc:347] Connected to a master server at 127.27.84.254:33643
I20260808 15:48:27.748473 31250 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:27.749130 31250 heartbeater.cc:511] Master 127.27.84.254:33643 requested a full tablet report, sending...
I20260808 15:48:27.750514 30706 ts_manager.cc:194] Registered new tserver with Master: 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799)
I20260808 15:48:27.751380 30706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:59747
I20260808 15:48:27.752100 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 31044
I20260808 15:48:27.752318 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:27.752806 27987 external_mini_cluster.cc:1428] Setting key d60d08a0e51c175bd509ef4cda7b7d76
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:27.806703 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33643
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:28.164088 31256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:28.164387 31256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:28.164601 31256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:28.172407 31256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:28.172538 31256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:28.172735 31256 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260808 15:48:28.172825 31256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:48:28.172971 31256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:28.175786 31039 heartbeater.cc:503] Master 127.27.84.254:33643 was elected leader, sending a full tablet report...
I20260808 15:48:28.183852 31256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:33643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:28.185302 31256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:28.186862 31256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:28.201416 31262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:28.201382 31268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:28.201483 31264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:28.202419 31256 server_base.cc:1034] running on GCE node
I20260808 15:48:28.203155 31256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:28.204222 31256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:28.205474 31256 hybrid_clock.cc:648] HybridClock initialized: now 1786204108205429 us; error 36 us; skew 500 ppm
I20260808 15:48:28.205837 31256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:28 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204108, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:28.209827 31256 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:28.211540 31256 webserver.cc:460] Webserver started at http://127.27.84.195:45269/ using document root <none> and password file <none>
I20260808 15:48:28.212267 31256 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:28.212431 31256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:28.212788 31256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:28.215899 31256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "3f367f8f1f824626972b6f16680a369e"
format_stamp: "Formatted at 2026-08-08 15:48:28 on dist-test-slave-twwt"
server_key: "61fa6c8a7fee76b4744eb1f7bc588ea1"
server_key_iv: "652fd1014b5196383d1dc79975d4f066"
server_key_version: "encryptionkey@0"
I20260808 15:48:28.216696 31256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "3f367f8f1f824626972b6f16680a369e"
format_stamp: "Formatted at 2026-08-08 15:48:28 on dist-test-slave-twwt"
server_key: "61fa6c8a7fee76b4744eb1f7bc588ea1"
server_key_iv: "652fd1014b5196383d1dc79975d4f066"
server_key_version: "encryptionkey@0"
I20260808 15:48:28.222288 31256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:48:28.225951 31278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:28.227455 31256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:28.227646 31256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "3f367f8f1f824626972b6f16680a369e"
format_stamp: "Formatted at 2026-08-08 15:48:28 on dist-test-slave-twwt"
server_key: "61fa6c8a7fee76b4744eb1f7bc588ea1"
server_key_iv: "652fd1014b5196383d1dc79975d4f066"
server_key_version: "encryptionkey@0"
I20260808 15:48:28.227914 31256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:28.251648 31256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:28.256219 31256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:28.256548 31256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:28.259423 31256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:28.259580 31256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:28.259720 31256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:28.259850 31256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:48:28 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204108, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:28.287055 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.259977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:47437 (local address 127.27.84.254:33643)
0808 15:48:28.260768 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.260774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.261919 (+ 1145us) server_negotiation.cc:407] Connection header received
0808 15:48:28.265262 (+ 3343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.265271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.265403 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.265535 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:28.268084 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.269080 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.272750 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.273415 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.273561 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.278396 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:28.278402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:28.278405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:28.278439 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:28.283075 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.283869 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.283878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.283884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.284091 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.284554 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.284562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.284568 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.285182 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:28.285480 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.286139 (+ 659us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.286468 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":525,"threads_started":1}
I20260808 15:48:28.287828 31293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.260728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.27.84.195:47437)
0808 15:48:28.261708 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.261773 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.264740 (+ 2967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.265882 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.265900 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.266749 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:28.267784 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.267824 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.269324 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.269334 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.272531 (+ 3197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.272558 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.273935 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.273943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.274493 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.275547 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:28.275578 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:28.278134 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:28.283263 (+ 5129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:28.283270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:28.283284 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:28.283715 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:28.284225 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:28.284228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:28.284230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:28.284427 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:28.285317 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:28.285323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:28.285676 (+ 353us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.285942 (+ 266us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.286341 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":367,"threads_started":1}
I20260808 15:48:28.308790 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.293591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.27.84.195:46787)
0808 15:48:28.294377 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.294400 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.294591 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.295021 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.295026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.295850 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:28.296315 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.296336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.297679 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.297683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.299686 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.299710 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.300715 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.300722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.300919 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.301573 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:28.301590 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:28.302436 (+ 846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:28.305991 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:28.305998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:28.306002 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:28.306412 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:28.306873 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:28.306882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:28.306885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:28.307120 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:28.307930 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:28.307934 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:28.308093 (+ 159us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.308211 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.308464 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":395,"threads_started":1}
I20260808 15:48:28.309206 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.293592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:46787 (local address 127.27.84.254:33643)
0808 15:48:28.293814 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.293823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.294522 (+ 699us) server_negotiation.cc:407] Connection header received
0808 15:48:28.294767 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.294775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.294871 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.294991 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:28.296476 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.297507 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.299908 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.300493 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.300626 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.302637 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:28.302644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:28.302649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:28.302687 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:28.305862 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.306535 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.306541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.306544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.306750 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.307270 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.307277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.307284 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.307768 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:28.307996 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.308373 (+ 377us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.308691 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:48:28.321110 31256 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:37147
I20260808 15:48:28.321146 31472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:37147 every 8 connection(s)
I20260808 15:48:28.322512 31256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:28.324826 31473 heartbeater.cc:347] Connected to a master server at 127.27.84.254:33643
I20260808 15:48:28.325042 31473 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:28.325675 31473 heartbeater.cc:511] Master 127.27.84.254:33643 requested a full tablet report, sending...
I20260808 15:48:28.326090 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 31256
I20260808 15:48:28.326272 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:28.326867 27987 external_mini_cluster.cc:1428] Setting key 4bd046a055c45c9e5e649bdd9672a48b
I20260808 15:48:28.327462 30706 ts_manager.cc:194] Registered new tserver with Master: 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:28.328713 30706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:47437
I20260808 15:48:28.330191 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:48:28 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:48:28 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:28.385133 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.371322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.27.84.254:33643)
0808 15:48:28.371510 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.371515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.371929 (+ 414us) server_negotiation.cc:407] Connection header received
0808 15:48:28.372101 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.372105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.372173 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.372267 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:28.373766 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.374484 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.375719 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.376128 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.376222 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.380142 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:28.380146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:28.380149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:28.380182 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:28.382923 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.383516 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.383519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.383522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.383649 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.383995 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.383998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.384000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.384274 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:28.384468 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.384696 (+ 228us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.384866 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:48:28.390054 30704 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:28.397545 30704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:28.424825 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.411018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.0.0.1:47190)
0808 15:48:28.412907 (+ 1889us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.412943 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.413184 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.414024 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.414038 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.414080 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.414593 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.414632 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.417478 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.417484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.419909 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.419927 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.421982 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.421988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.423885 (+ 1897us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.423947 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.424109 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.424314 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"mutex_wait_us":1569,"thread_start_us":1467,"threads_started":1}
I20260808 15:48:28.425799 31488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.411686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:37147 (local address 127.0.0.1:54180)
0808 15:48:28.414482 (+ 2796us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.414506 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.414759 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.416401 (+ 1642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.416406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.416431 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.416857 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.416869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.420141 (+ 3272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.420146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.422292 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.422307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.424552 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.424559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.424770 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.424817 (+ 47us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.425171 (+ 354us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.425344 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2645,"mutex_wait_us":41,"thread_start_us":2593,"threads_started":1}
I20260808 15:48:28.426544 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.411315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.27.84.193:46045)
0808 15:48:28.412163 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.412173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.413071 (+ 898us) server_negotiation.cc:407] Connection header received
0808 15:48:28.413436 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.413448 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.413744 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.415381 (+ 1637us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.415624 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.417308 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.420168 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.421738 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.422230 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.422483 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.426053 (+ 3570us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.426243 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":479,"threads_started":1}
I20260808 15:48:28.427574 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.412354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54180 (local address 127.27.84.195:37147)
0808 15:48:28.413971 (+ 1617us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.413979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.414675 (+ 696us) server_negotiation.cc:407] Connection header received
0808 15:48:28.414946 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.414954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.415219 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.416087 (+ 868us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.417085 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.419979 (+ 2894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.422486 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.424040 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.424163 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.424398 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.426124 (+ 1726us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.426292 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1400,"thread_start_us":1684,"threads_started":1}
I20260808 15:48:28.430812 31351 tablet_service.cc:1467] Processing CreateTablet for tablet 1acd5f83fe8a45389b99f591e0360a0e (DEFAULT_TABLE table=test-table [id=8d1738928473437fbb7335550092b7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:28.430821 30926 tablet_service.cc:1467] Processing CreateTablet for tablet 1acd5f83fe8a45389b99f591e0360a0e (DEFAULT_TABLE table=test-table [id=8d1738928473437fbb7335550092b7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:28.432335 30926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acd5f83fe8a45389b99f591e0360a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.432317 31351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acd5f83fe8a45389b99f591e0360a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.433694 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.413482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37799 (local address 127.0.0.1:46896)
0808 15:48:28.425077 (+ 11595us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.425094 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.425187 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.427098 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.427103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.427129 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.427303 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.427312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.429195 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.429200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.430999 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.431015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.432810 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.432816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.433100 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.433136 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.433242 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.433398 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11502,"thread_start_us":12412,"threads_started":1}
I20260808 15:48:28.434276 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.413799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.27.84.194:37799)
0808 15:48:28.425759 (+ 11960us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.425783 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.425817 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:48:28.426579 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.426609 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.426886 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.427081 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.427494 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.429030 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.431206 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.432566 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.432735 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.432960 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.433320 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.433500 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11764,"thread_start_us":11564,"threads_started":1}
I20260808 15:48:28.442896 31144 tablet_service.cc:1467] Processing CreateTablet for tablet 1acd5f83fe8a45389b99f591e0360a0e (DEFAULT_TABLE table=test-table [id=8d1738928473437fbb7335550092b7ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:28.444473 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1acd5f83fe8a45389b99f591e0360a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.444996 31493 tablet_bootstrap.cc:492] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Bootstrap starting.
I20260808 15:48:28.447252 31494 tablet_bootstrap.cc:492] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Bootstrap starting.
I20260808 15:48:28.449170 31493 tablet_bootstrap.cc:654] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.450645 31493 log.cc:824] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:28.451138 31494 tablet_bootstrap.cc:654] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.452651 31494 log.cc:824] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:28.454694 31493 tablet_bootstrap.cc:492] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: No bootstrap required, opened a new log
I20260808 15:48:28.455032 31493 ts_tablet_manager.cc:1397] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260808 15:48:28.457374 31494 tablet_bootstrap.cc:492] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: No bootstrap required, opened a new log
I20260808 15:48:28.457666 31494 ts_tablet_manager.cc:1397] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260808 15:48:28.459471 31497 tablet_bootstrap.cc:492] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Bootstrap starting.
I20260808 15:48:28.463501 31497 tablet_bootstrap.cc:654] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.464699 31497 log.cc:824] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:28.466270 31493 raft_consensus.cc:348] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.466780 31493 raft_consensus.cc:374] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.466881 31493 raft_consensus.cc:729] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.467543 31493 consensus_queue.cc:260] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.468354 31497 tablet_bootstrap.cc:492] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: No bootstrap required, opened a new log
I20260808 15:48:28.468676 31497 ts_tablet_manager.cc:1397] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:48:28.468513 31494 raft_consensus.cc:348] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.469072 31494 raft_consensus.cc:374] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.469202 31494 raft_consensus.cc:729] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f367f8f1f824626972b6f16680a369e, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.470132 31494 consensus_queue.cc:260] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.475925 31493 ts_tablet_manager.cc:1428] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent starting tablet: real 0.021s user 0.018s sys 0.002s
I20260808 15:48:28.480131 31494 ts_tablet_manager.cc:1428] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Time spent starting tablet: real 0.022s user 0.020s sys 0.002s
I20260808 15:48:28.480472 31473 heartbeater.cc:503] Master 127.27.84.254:33643 was elected leader, sending a full tablet report...
I20260808 15:48:28.481060 31497 raft_consensus.cc:348] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.481534 31497 raft_consensus.cc:374] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.481642 31497 raft_consensus.cc:729] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.482395 31497 consensus_queue.cc:260] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.484336 31250 heartbeater.cc:503] Master 127.27.84.254:33643 was elected leader, sending a full tablet report...
I20260808 15:48:28.484920 31497 ts_tablet_manager.cc:1428] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260808 15:48:28.498273 31251 tablet.cc:2369] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:28.508607 31503 raft_consensus.cc:482] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:28.508931 31503 raft_consensus.cc:504] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.511027 31503 leader_election.cc:288] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:28.521164 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.511750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:59389 (local address 127.27.84.195:37147)
0808 15:48:28.512014 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.512024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.513066 (+ 1042us) server_negotiation.cc:407] Connection header received
0808 15:48:28.513237 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.513245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.513326 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.513420 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.514248 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.515434 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.518542 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.519880 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.520052 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.520133 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.520785 (+ 652us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.520916 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260808 15:48:28.521164 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.512018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:59877 (local address 127.27.84.193:46045)
0808 15:48:28.512316 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.512324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.512780 (+ 456us) server_negotiation.cc:407] Connection header received
0808 15:48:28.513059 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.513065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.513169 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.513266 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.514248 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.515434 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.518546 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.519866 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.520052 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.520133 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.520748 (+ 615us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.520897 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 15:48:28.521572 31504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.511430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.27.84.194:59877)
0808 15:48:28.512645 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.512662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.512895 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.513320 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.513325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.513345 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.514094 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.514105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.515589 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.515595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.518334 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.518351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.520183 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.520187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.520479 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.520585 (+ 106us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.520688 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.520943 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":245,"thread_start_us":760,"threads_started":1}
I20260808 15:48:28.521685 31506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.511479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:37147 (local address 127.27.84.194:59389)
0808 15:48:28.512973 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.512988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.513104 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.513446 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.513453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.513475 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.514016 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.514037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.515589 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.515595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.518335 (+ 2740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.518351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.520180 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.520187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.520475 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.520587 (+ 112us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.520688 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.520943 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1312,"mutex_wait_us":302,"thread_start_us":1070,"threads_started":1}
I20260808 15:48:28.522739 30967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acd5f83fe8a45389b99f591e0360a0e" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" is_pre_election: true
I20260808 15:48:28.522742 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acd5f83fe8a45389b99f591e0360a0e" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e" is_pre_election: true
I20260808 15:48:28.523270 30967 raft_consensus.cc:2454] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 0.
I20260808 15:48:28.523272 31400 raft_consensus.cc:2454] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 0.
I20260808 15:48:28.523948 31068 leader_election.cc:302] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 4c3a7d68ace9422fb626b4e15ebdc9df, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:28.524395 31503 raft_consensus.cc:2781] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:28.524546 31503 raft_consensus.cc:482] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:28.524665 31503 raft_consensus.cc:3037] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:28.527240 31503 raft_consensus.cc:504] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.527989 31503 leader_election.cc:288] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 election: Requested vote from peers 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:28.528451 30967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acd5f83fe8a45389b99f591e0360a0e" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df"
I20260808 15:48:28.528496 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1acd5f83fe8a45389b99f591e0360a0e" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e"
I20260808 15:48:28.528692 31400 raft_consensus.cc:3037] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:28.528685 30967 raft_consensus.cc:3037] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:28.532104 30967 raft_consensus.cc:2454] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 1.
I20260808 15:48:28.532104 31400 raft_consensus.cc:2454] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 1.
I20260808 15:48:28.532557 31066 leader_election.cc:302] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 4c3a7d68ace9422fb626b4e15ebdc9df, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:28.532940 31503 raft_consensus.cc:2781] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:28.533429 31503 raft_consensus.cc:686] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 1 LEADER]: Becoming Leader. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Running, Role: LEADER
I20260808 15:48:28.533921 31503 consensus_queue.cc:237] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.539878 30704 catalog_manager.cc:5495] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "78ca76d31d8e40b89806a9cb624bb630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } health_report { overall_health: UNKNOWN } } }
W20260808 15:48:28.574220 31474 tablet.cc:2369] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 08 15:48:28 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:28.607101 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.593259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.27.84.254:33643)
0808 15:48:28.593502 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.593510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.594107 (+ 597us) server_negotiation.cc:407] Connection header received
0808 15:48:28.594252 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.594256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.594332 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.594429 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:28.595795 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.596557 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.597817 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.598249 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.598350 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.601637 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:28.601641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:28.601645 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:28.601677 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:28.604419 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.605076 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.605080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.605082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.605208 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:28.605621 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:28.605624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:28.605626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:28.605877 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:28.605986 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.606489 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.606782 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":92}
I20260808 15:48:28.607192 31516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.593282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33643 (local address 127.0.0.1:37298)
0808 15:48:28.593983 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.594000 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.594139 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.594489 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.594493 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.595331 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:28.595616 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.595625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.596691 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.596695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.597660 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.597671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.598433 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.598438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.598548 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.599245 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:28.599261 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:28.601442 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:28.604543 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:28.604554 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:28.604560 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:28.604967 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:28.605344 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:28.605348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:28.605350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:28.605504 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:28.606052 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:28.606059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:28.606228 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.606383 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.606679 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":382,"threads_started":1}
I20260808 15:48:28.612839 30704 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:37298:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:48:28.613871 30704 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:28.621098 31144 tablet_service.cc:1467] Processing CreateTablet for tablet 385b732964464060858b861fa636f3c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11da098d76cd4fc4827b2ca414f80e7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:48:28.621276 31351 tablet_service.cc:1467] Processing CreateTablet for tablet 385b732964464060858b861fa636f3c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11da098d76cd4fc4827b2ca414f80e7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:48:28.621140 30926 tablet_service.cc:1467] Processing CreateTablet for tablet 385b732964464060858b861fa636f3c1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11da098d76cd4fc4827b2ca414f80e7a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:48:28.622030 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b732964464060858b861fa636f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.622112 30926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b732964464060858b861fa636f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.622112 31351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385b732964464060858b861fa636f3c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.628209 31497 tablet_bootstrap.cc:492] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: Bootstrap starting.
I20260808 15:48:28.629751 31493 tablet_bootstrap.cc:492] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Bootstrap starting.
I20260808 15:48:28.629951 31497 tablet_bootstrap.cc:654] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.630239 31494 tablet_bootstrap.cc:492] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: Bootstrap starting.
I20260808 15:48:28.631721 31493 tablet_bootstrap.cc:654] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.632134 31494 tablet_bootstrap.cc:654] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.633169 31497 tablet_bootstrap.cc:492] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: No bootstrap required, opened a new log
I20260808 15:48:28.633421 31497 ts_tablet_manager.cc:1397] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 15:48:28.634512 31494 tablet_bootstrap.cc:492] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: No bootstrap required, opened a new log
I20260808 15:48:28.634353 31497 raft_consensus.cc:348] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.634654 31493 tablet_bootstrap.cc:492] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: No bootstrap required, opened a new log
I20260808 15:48:28.634666 31497 raft_consensus.cc:374] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.634712 31494 ts_tablet_manager.cc:1397] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260808 15:48:28.634761 31497 raft_consensus.cc:729] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.634903 31493 ts_tablet_manager.cc:1397] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260808 15:48:28.635007 31497 consensus_queue.cc:260] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.635465 31494 raft_consensus.cc:348] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.635476 31503 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.635697 31494 raft_consensus.cc:374] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.635752 31503 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.635792 31494 raft_consensus.cc:729] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f367f8f1f824626972b6f16680a369e, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.635715 31493 raft_consensus.cc:348] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.636109 31494 consensus_queue.cc:260] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.636298 31493 raft_consensus.cc:374] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.636405 31493 raft_consensus.cc:729] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.636586 31500 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.636960 31500 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.637058 31493 consensus_queue.cc:260] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.637458 31494 ts_tablet_manager.cc:1428] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:28.637562 31499 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.637964 31499 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.638851 31493 ts_tablet_manager.cc:1428] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260808 15:48:28.639422 31497 ts_tablet_manager.cc:1428] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20260808 15:48:28.668345 31040 tablet.cc:2369] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:28.703482 31499 raft_consensus.cc:482] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:28.703790 31499 raft_consensus.cc:504] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.705449 31499 leader_election.cc:288] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:28.713155 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.705875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:34217 (local address 127.27.84.194:37799)
0808 15:48:28.706159 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.706169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.706924 (+ 755us) server_negotiation.cc:407] Connection header received
0808 15:48:28.707148 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.707152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.707216 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.707336 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.708073 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.709131 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.711139 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.712070 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.712269 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.712365 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.712695 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.712862 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:48:28.713177 31489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.706034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:35325 (local address 127.27.84.195:37147)
0808 15:48:28.706258 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.706267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.707010 (+ 743us) server_negotiation.cc:407] Connection header received
0808 15:48:28.707223 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.707227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.707286 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.707402 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:28.708336 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.709301 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.711112 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.712015 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.712201 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.712286 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.712720 (+ 434us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.712868 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:48:28.713425 31524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.705781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37799 (local address 127.27.84.193:34217)
0808 15:48:28.706774 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.706792 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.706995 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.707355 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.707360 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.707385 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.707808 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.707837 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.709273 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.709280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.710971 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.710984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.712305 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.712311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.712513 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.712544 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.712664 (+ 120us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.712927 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":90,"thread_start_us":615,"threads_started":1}
I20260808 15:48:28.713792 31526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.705905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:37147 (local address 127.27.84.193:35325)
0808 15:48:28.706913 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.706932 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.707043 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.707417 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.707665 (+ 248us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.707690 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:28.708132 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.708148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.709457 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.709463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.710942 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.710959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.712257 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.712307 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.712527 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.712557 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.712671 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.712927 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":72,"thread_start_us":616,"threads_started":1}
I20260808 15:48:28.714432 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b732964464060858b861fa636f3c1" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e" is_pre_election: true
I20260808 15:48:28.714720 31400 raft_consensus.cc:2454] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 0.
I20260808 15:48:28.714501 31169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b732964464060858b861fa636f3c1" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca76d31d8e40b89806a9cb624bb630" is_pre_election: true
I20260808 15:48:28.715041 31169 raft_consensus.cc:2454] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 0.
I20260808 15:48:28.715278 30858 leader_election.cc:302] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 4c3a7d68ace9422fb626b4e15ebdc9df; no voters:
I20260808 15:48:28.715726 31499 raft_consensus.cc:2781] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:28.715874 31499 raft_consensus.cc:482] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:28.715981 31499 raft_consensus.cc:3037] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:28.718242 31499 raft_consensus.cc:504] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.718902 31499 leader_election.cc:288] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 election: Requested vote from peers 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:28.719444 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b732964464060858b861fa636f3c1" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e"
I20260808 15:48:28.719466 31169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385b732964464060858b861fa636f3c1" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca76d31d8e40b89806a9cb624bb630"
I20260808 15:48:28.719661 31169 raft_consensus.cc:3037] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:28.719684 31400 raft_consensus.cc:3037] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:28.722796 31169 raft_consensus.cc:2454] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 1.
I20260808 15:48:28.722780 31400 raft_consensus.cc:2454] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 1.
I20260808 15:48:28.723266 30860 leader_election.cc:302] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c3a7d68ace9422fb626b4e15ebdc9df, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:28.723600 31499 raft_consensus.cc:2781] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:28.724074 31499 raft_consensus.cc:686] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 LEADER]: Becoming Leader. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Running, Role: LEADER
I20260808 15:48:28.724565 31499 consensus_queue.cc:237] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.727978 31530 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: TxnStatusTablet state changed. Reason: New leader 4c3a7d68ace9422fb626b4e15ebdc9df. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.728368 31530 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:48:28.729441 31533 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:48:28.729720 31533 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:48:28.730057 30706 catalog_manager.cc:5495] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193). New cstate: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } health_report { overall_health: UNKNOWN } } }
I20260808 15:48:28.770732 31516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.764290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.0.0.1:47196)
0808 15:48:28.764626 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.764642 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.764837 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.765368 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.765373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.765386 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:28.765609 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.765623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.766822 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.766828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.768081 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.768095 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.768952 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.768958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.769146 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.769207 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.770107 (+ 900us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.770131 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.770206 (+ 75us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.770373 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 15:48:28.770751 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.764282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.27.84.193:46045)
0808 15:48:28.764502 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.764508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.764752 (+ 244us) server_negotiation.cc:407] Connection header received
0808 15:48:28.765087 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.765091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.765179 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.765352 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:28.765776 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.766665 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.768227 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.768742 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.768950 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.769434 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.769983 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.770162 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.770224 (+ 62us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.770318 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:48:28.779210 31400 raft_consensus.cc:1261] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Refusing update from remote peer 4c3a7d68ace9422fb626b4e15ebdc9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:28.779225 31169 raft_consensus.cc:1261] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Refusing update from remote peer 4c3a7d68ace9422fb626b4e15ebdc9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:28.780272 31530 consensus_queue.cc:1035] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:28.780639 31499 consensus_queue.cc:1035] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:28.790215 31503 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: TxnStatusTablet state changed. Reason: New leader 4c3a7d68ace9422fb626b4e15ebdc9df. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.790596 31503 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.794276 31538 mvcc.cc:205] Tried to move back new op lower bound from 7316292029548359680 to 7316292029340102656. Current Snapshot: MvccSnapshot[applied={T|T < 7316292029548359680}]
I20260808 15:48:28.795893 31540 mvcc.cc:205] Tried to move back new op lower bound from 7316292029548359680 to 7316292029340102656. Current Snapshot: MvccSnapshot[applied={T|T < 7316292029548359680}]
I20260808 15:48:28.797502 31503 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.797823 31503 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 78ca76d31d8e40b89806a9cb624bb630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.801388 31530 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.801682 31530 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:48:28.804796 31500 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: TxnStatusTablet state changed. Reason: New leader 4c3a7d68ace9422fb626b4e15ebdc9df. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.805081 31500 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.808552 31499 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.808948 31499 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 4c3a7d68ace9422fb626b4e15ebdc9df: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:48:28.812750 31500 tablet_replica.cc:439] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } } }
I20260808 15:48:28.813012 31500 tablet_replica.cc:442] T 385b732964464060858b861fa636f3c1 P 3f367f8f1f824626972b6f16680a369e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:48:28.820010 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.808675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.27.84.194:37799)
0808 15:48:28.808874 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.808881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.809351 (+ 470us) server_negotiation.cc:407] Connection header received
0808 15:48:28.809447 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.809450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.809528 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.809703 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:28.810148 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.811384 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.816585 (+ 5201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.817538 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.817705 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.818651 (+ 946us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.819322 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.819508 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.819595 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.819690 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:48:28.839054 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.832200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36587 (local address 127.27.84.193:46045)
0808 15:48:28.832517 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.832526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.833090 (+ 564us) server_negotiation.cc:407] Connection header received
0808 15:48:28.833292 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.833296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.833374 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.833448 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:28.834080 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.834996 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.837108 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.837583 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.837682 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.838296 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.838413 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.838481 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.838735 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.838804 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:48:28.839334 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.832195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.27.84.194:36587)
0808 15:48:28.832974 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.832996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.833158 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.833501 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.833506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.833520 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:28.833898 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.833914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.835174 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.835180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.836942 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.836960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.837780 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.837788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.837983 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.838066 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.838521 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.838534 (+ 13us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.838645 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.838803 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":387,"threads_started":1}
I20260808 15:48:28.850106 31400 raft_consensus.cc:1261] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Refusing update from remote peer 78ca76d31d8e40b89806a9cb624bb630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:28.850664 30967 raft_consensus.cc:1261] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Refusing update from remote peer 78ca76d31d8e40b89806a9cb624bb630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:28.850772 31503 consensus_queue.cc:1035] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:28.851747 31503 consensus_queue.cc:1035] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:28.862211 31541 mvcc.cc:205] Tried to move back new op lower bound from 7316292029843701760 to 7316292028559261696. Current Snapshot: MvccSnapshot[applied={T|T < 7316292029843701760}]
I20260808 15:48:28.892019 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.885011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:32895 (local address 127.27.84.194:37799)
0808 15:48:28.885218 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:28.885226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:28.885984 (+ 758us) server_negotiation.cc:407] Connection header received
0808 15:48:28.886284 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:28.886288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:28.886353 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:28.886451 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:28.887119 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.887957 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.890036 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.890552 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.890750 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.891306 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.891408 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.891510 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:28.891634 (+ 124us) server_negotiation.cc:299] Negotiation successful
0808 15:48:28.891716 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:48:28.892488 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:28.885027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37799 (local address 127.27.84.193:32895)
0808 15:48:28.885835 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:28.885868 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:28.886115 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:28.886472 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:28.886481 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:28.886493 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:28.886923 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.886934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.888081 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.888086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.889899 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:28.889919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:28.890818 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:28.890826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:28.891017 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:28.891077 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:28.891516 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:28.891530 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 15:48:28.891613 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:48:28.891768 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":411,"threads_started":1}
I20260808 15:48:28.942996 30706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:28.943348 30706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:28.948076 30706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Sending DeleteTablet for 3 replicas of tablet 1acd5f83fe8a45389b99f591e0360a0e
I20260808 15:48:28.949654 31351 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acd5f83fe8a45389b99f591e0360a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:28 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:54180
I20260808 15:48:28.949774 30926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acd5f83fe8a45389b99f591e0360a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:28 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260808 15:48:28.951018 31144 tablet_service.cc:1514] Processing DeleteTablet for tablet 1acd5f83fe8a45389b99f591e0360a0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:28 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:46896
I20260808 15:48:28.952188 30706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:28.954010 30706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:28.956426 31577 tablet_replica.cc:330] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: stopping tablet replica
I20260808 15:48:28.957001 31577 raft_consensus.cc:2229] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:28.957816 31577 raft_consensus.cc:2258] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:28.963915 31350 tablet_service.cc:1467] Processing CreateTablet for tablet 1948b634183140f29e7137ca4dbb41a1 (DEFAULT_TABLE table=test-table [id=c72acb4f480c4524a2886c21189677cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:28.964494 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1948b634183140f29e7137ca4dbb41a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.964998 31572 tablet_replica.cc:330] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: stopping tablet replica
I20260808 15:48:28.965694 31572 raft_consensus.cc:2229] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:28.966205 31572 raft_consensus.cc:2258] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:28.967232 31576 tablet_replica.cc:330] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: stopping tablet replica
I20260808 15:48:28.967605 31144 tablet_service.cc:1467] Processing CreateTablet for tablet 1948b634183140f29e7137ca4dbb41a1 (DEFAULT_TABLE table=test-table [id=c72acb4f480c4524a2886c21189677cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:28.967732 31576 raft_consensus.cc:2229] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:28.968146 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1948b634183140f29e7137ca4dbb41a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.968264 31576 raft_consensus.cc:2258] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:28.970468 31576 ts_tablet_manager.cc:1905] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:28.972141 30923 tablet_service.cc:1467] Processing CreateTablet for tablet 1948b634183140f29e7137ca4dbb41a1 (DEFAULT_TABLE table=test-table [id=c72acb4f480c4524a2886c21189677cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:28.972597 31577 ts_tablet_manager.cc:1905] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:28.972757 30923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1948b634183140f29e7137ca4dbb41a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:28.980608 31577 ts_tablet_manager.cc:1918] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:28.980820 31577 log.cc:1196] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/1acd5f83fe8a45389b99f591e0360a0e
I20260808 15:48:28.981472 31577 ts_tablet_manager.cc:1939] T 1acd5f83fe8a45389b99f591e0360a0e P 78ca76d31d8e40b89806a9cb624bb630: Deleting consensus metadata
I20260808 15:48:28.983645 30683 catalog_manager.cc:4841] TS 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799): tablet 1acd5f83fe8a45389b99f591e0360a0e (table test-table [id=8d1738928473437fbb7335550092b7ed]) successfully deleted
I20260808 15:48:28.984323 31497 tablet_bootstrap.cc:492] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Bootstrap starting.
I20260808 15:48:28.984596 31572 ts_tablet_manager.cc:1905] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:28.984658 31494 tablet_bootstrap.cc:492] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Bootstrap starting.
I20260808 15:48:28.986119 31497 tablet_bootstrap.cc:654] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.986742 31494 tablet_bootstrap.cc:654] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:28.988610 31497 tablet_bootstrap.cc:492] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: No bootstrap required, opened a new log
I20260808 15:48:28.988798 31497 ts_tablet_manager.cc:1397] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260808 15:48:28.989686 31497 raft_consensus.cc:348] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.989944 31497 raft_consensus.cc:374] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.990020 31497 raft_consensus.cc:729] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.990270 31497 consensus_queue.cc:260] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.990936 31497 ts_tablet_manager.cc:1428] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:48:28.992738 31572 ts_tablet_manager.cc:1918] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:28.992944 31572 log.cc:1196] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/1acd5f83fe8a45389b99f591e0360a0e
I20260808 15:48:28.993539 31572 ts_tablet_manager.cc:1939] T 1acd5f83fe8a45389b99f591e0360a0e P 3f367f8f1f824626972b6f16680a369e: Deleting consensus metadata
I20260808 15:48:28.995672 30681 catalog_manager.cc:4841] TS 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147): tablet 1acd5f83fe8a45389b99f591e0360a0e (table test-table [id=8d1738928473437fbb7335550092b7ed]) successfully deleted
I20260808 15:48:28.996789 31494 tablet_bootstrap.cc:492] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: No bootstrap required, opened a new log
I20260808 15:48:28.996984 31494 ts_tablet_manager.cc:1397] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260808 15:48:28.997763 31494 raft_consensus.cc:348] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.998061 31494 raft_consensus.cc:374] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:28.998149 31494 raft_consensus.cc:729] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f367f8f1f824626972b6f16680a369e, State: Initialized, Role: FOLLOWER
I20260808 15:48:28.998399 31494 consensus_queue.cc:260] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:28.999137 31494 ts_tablet_manager.cc:1428] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 15:48:29.001365 31576 ts_tablet_manager.cc:1918] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:29.001571 31576 log.cc:1196] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/1acd5f83fe8a45389b99f591e0360a0e
I20260808 15:48:29.002179 31576 ts_tablet_manager.cc:1939] T 1acd5f83fe8a45389b99f591e0360a0e P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting consensus metadata
I20260808 15:48:29.004084 30683 catalog_manager.cc:4841] TS 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045): tablet 1acd5f83fe8a45389b99f591e0360a0e (table test-table [id=8d1738928473437fbb7335550092b7ed]) successfully deleted
I20260808 15:48:29.004664 31493 tablet_bootstrap.cc:492] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Bootstrap starting.
I20260808 15:48:29.006537 31493 tablet_bootstrap.cc:654] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:29.009194 31493 tablet_bootstrap.cc:492] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: No bootstrap required, opened a new log
I20260808 15:48:29.009377 31493 ts_tablet_manager.cc:1397] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260808 15:48:29.010142 31493 raft_consensus.cc:348] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:29.010408 31493 raft_consensus.cc:374] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:29.010491 31493 raft_consensus.cc:729] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Initialized, Role: FOLLOWER
I20260808 15:48:29.010803 31493 consensus_queue.cc:260] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:29.011590 31493 ts_tablet_manager.cc:1428] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260808 15:48:29.166384 31500 raft_consensus.cc:482] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:29.166683 31500 raft_consensus.cc:504] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:29.168874 31500 leader_election.cc:288] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045), 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799)
I20260808 15:48:29.198392 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.182745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51563 (local address 127.27.84.194:37799)
0808 15:48:29.182956 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:29.182962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:29.190102 (+ 7140us) server_negotiation.cc:407] Connection header received
0808 15:48:29.190458 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:29.190464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:29.190567 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:29.190659 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:29.191710 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.192985 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.195871 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.197161 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.197308 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.197385 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:29.197949 (+ 564us) server_negotiation.cc:299] Negotiation successful
0808 15:48:29.198090 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:48:29.199213 31584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.183124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37799 (local address 127.27.84.195:51563)
0808 15:48:29.189976 (+ 6852us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:29.190004 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:29.190232 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:29.190937 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:29.190944 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:29.190973 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:29.191494 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.191513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.193216 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.193221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.195670 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.195686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.197555 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.197561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.197742 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.197779 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:48:29.198679 (+ 900us) client_negotiation.cc:240] Negotiation successful
0808 15:48:29.198837 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6489,"mutex_wait_us":28,"thread_start_us":6609,"threads_started":1}
I20260808 15:48:29.199846 31169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1948b634183140f29e7137ca4dbb41a1" candidate_uuid: "3f367f8f1f824626972b6f16680a369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca76d31d8e40b89806a9cb624bb630" is_pre_election: true
I20260808 15:48:29.200119 31169 raft_consensus.cc:2454] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f367f8f1f824626972b6f16680a369e in term 0.
I20260808 15:48:29.200663 31286 leader_election.cc:302] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:29.201162 31500 raft_consensus.cc:2781] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:29.201299 31500 raft_consensus.cc:482] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:29.201398 31500 raft_consensus.cc:3037] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:29.202026 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.169942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:36105 (local address 127.27.84.193:46045)
0808 15:48:29.170165 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:29.170172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:29.181936 (+ 11764us) server_negotiation.cc:407] Connection header received
0808 15:48:29.182186 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:29.182194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:29.182282 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:29.182396 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:29.189611 (+ 7215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.190997 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.198841 (+ 7844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.200196 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.200374 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.200450 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:29.201541 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0808 15:48:29.201691 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:48:29.203236 31583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.169520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.27.84.195:36105)
0808 15:48:29.181770 (+ 12250us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:29.181804 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:29.182035 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:29.186734 (+ 4699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:29.186741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:29.186779 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:29.187181 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.187198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.191250 (+ 4052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.191258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.193668 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.193685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.200984 (+ 7299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.200992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.201186 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.201221 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:48:29.202645 (+ 1424us) client_negotiation.cc:240] Negotiation successful
0808 15:48:29.202813 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12074,"mutex_wait_us":1,"thread_start_us":12619,"threads_started":1}
I20260808 15:48:29.204659 30967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1948b634183140f29e7137ca4dbb41a1" candidate_uuid: "3f367f8f1f824626972b6f16680a369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" is_pre_election: true
I20260808 15:48:29.204931 30967 raft_consensus.cc:2454] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f367f8f1f824626972b6f16680a369e in term 0.
I20260808 15:48:29.206449 31500 raft_consensus.cc:504] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:29.207116 31500 leader_election.cc:288] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [CANDIDATE]: Term 1 election: Requested vote from peers 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045), 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799)
I20260808 15:48:29.208115 31169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1948b634183140f29e7137ca4dbb41a1" candidate_uuid: "3f367f8f1f824626972b6f16680a369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca76d31d8e40b89806a9cb624bb630"
I20260808 15:48:29.208371 31169 raft_consensus.cc:3037] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:29.211763 31169 raft_consensus.cc:2454] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f367f8f1f824626972b6f16680a369e in term 1.
I20260808 15:48:29.212633 30967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1948b634183140f29e7137ca4dbb41a1" candidate_uuid: "3f367f8f1f824626972b6f16680a369e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df"
I20260808 15:48:29.212934 30967 raft_consensus.cc:3037] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:29.215409 31286 leader_election.cc:302] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:29.216125 31500 raft_consensus.cc:2781] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:29.217278 30967 raft_consensus.cc:2454] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f367f8f1f824626972b6f16680a369e in term 1.
I20260808 15:48:29.219688 31500 raft_consensus.cc:686] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 1 LEADER]: Becoming Leader. State: Replica: 3f367f8f1f824626972b6f16680a369e, State: Running, Role: LEADER
I20260808 15:48:29.220184 31500 consensus_queue.cc:237] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:29.224809 30706 catalog_manager.cc:5495] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f367f8f1f824626972b6f16680a369e (127.27.84.195). New cstate: current_term: 1 leader_uuid: "3f367f8f1f824626972b6f16680a369e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } health_report { overall_health: HEALTHY } } }
I20260808 15:48:29.272460 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.259128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.27.84.195:37147)
0808 15:48:29.265043 (+ 5915us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:29.265049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:29.265075 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:48:29.265186 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:29.265192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:29.265305 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:29.265509 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:29.266064 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.267288 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.269114 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.269907 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.270042 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.270525 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:29.271540 (+ 1015us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:29.271706 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:29.271924 (+ 218us) server_negotiation.cc:299] Negotiation successful
0808 15:48:29.272023 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5769,"thread_start_us":5586,"threads_started":1}
I20260808 15:48:29.291404 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.284389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:57421 (local address 127.27.84.193:46045)
0808 15:48:29.284602 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:29.284606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:29.285304 (+ 698us) server_negotiation.cc:407] Connection header received
0808 15:48:29.285464 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:29.285470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:29.285566 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:29.285668 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:29.286257 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.287494 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.289069 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.289880 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.290028 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.290569 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:29.290761 (+ 192us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:29.290875 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:29.291022 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 15:48:29.291114 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:48:29.291451 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.284427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.27.84.195:57421)
0808 15:48:29.285166 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:29.285182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:29.285325 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:29.285717 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:29.285722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:29.285735 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:29.286092 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.286102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.287659 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.287664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.288902 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.288916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.290120 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.290125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.290262 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.290380 (+ 118us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:29.290868 (+ 488us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:29.290877 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:48:29.290977 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:48:29.291089 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":406,"threads_started":1}
I20260808 15:48:29.300935 31169 raft_consensus.cc:1261] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Refusing update from remote peer 3f367f8f1f824626972b6f16680a369e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:29.301182 30967 raft_consensus.cc:1261] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Refusing update from remote peer 3f367f8f1f824626972b6f16680a369e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:29.301872 31585 consensus_queue.cc:1035] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:29.302397 31500 consensus_queue.cc:1035] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:29.341640 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.334330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:37147 (local address 127.27.84.193:43831)
0808 15:48:29.334550 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:29.334574 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:29.334717 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:29.335313 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:29.335318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:29.335329 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:29.335559 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.335572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.337158 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.337164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.338910 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:29.338926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.340301 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.340307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:29.340502 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.340552 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:29.340998 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:29.341012 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 15:48:29.341083 (+ 71us) client_negotiation.cc:240] Negotiation successful
0808 15:48:29.341238 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260808 15:48:29.342386 31588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:29.334420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:43831 (local address 127.27.84.195:37147)
0808 15:48:29.335017 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:29.335023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:29.335041 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:29.335105 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:29.335112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:29.335192 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:29.335297 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:29.335759 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.336977 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.339083 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:29.340059 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:29.340177 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:29.340721 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:29.340852 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:29.340945 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:29.341995 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0808 15:48:29.342082 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
W20260808 15:48:29.343535 31541 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 15:48:30.963660 30706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:30.963994 30706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:30.968937 30706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Sending DeleteTablet for 3 replicas of tablet 1948b634183140f29e7137ca4dbb41a1
I20260808 15:48:30.969878 30926 tablet_service.cc:1514] Processing DeleteTablet for tablet 1948b634183140f29e7137ca4dbb41a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:30 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260808 15:48:30.970160 31350 tablet_service.cc:1514] Processing DeleteTablet for tablet 1948b634183140f29e7137ca4dbb41a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:30 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:54180
I20260808 15:48:30.970173 31144 tablet_service.cc:1514] Processing DeleteTablet for tablet 1948b634183140f29e7137ca4dbb41a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:30 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:46896
I20260808 15:48:30.982646 31636 tablet_replica.cc:330] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: stopping tablet replica
I20260808 15:48:30.982966 31639 tablet_replica.cc:330] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: stopping tablet replica
I20260808 15:48:30.982944 31636 raft_consensus.cc:2229] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:30.983206 31636 raft_consensus.cc:2258] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:30.983260 31639 raft_consensus.cc:2229] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:30.984016 31639 raft_consensus.cc:2258] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:30.984488 31636 ts_tablet_manager.cc:1905] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:30.985241 31639 ts_tablet_manager.cc:1905] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:30.986843 31637 tablet_replica.cc:330] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: stopping tablet replica
I20260808 15:48:30.987248 31637 raft_consensus.cc:2229] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:30.987748 31637 raft_consensus.cc:2258] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:30.989329 31637 ts_tablet_manager.cc:1905] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:30.991649 31636 ts_tablet_manager.cc:1918] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:48:30.991811 31636 log.cc:1196] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/1948b634183140f29e7137ca4dbb41a1
I20260808 15:48:30.992316 31636 ts_tablet_manager.cc:1939] T 1948b634183140f29e7137ca4dbb41a1 P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting consensus metadata
I20260808 15:48:30.992457 31639 ts_tablet_manager.cc:1918] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:48:30.992600 31639 log.cc:1196] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/1948b634183140f29e7137ca4dbb41a1
I20260808 15:48:30.993101 31639 ts_tablet_manager.cc:1939] T 1948b634183140f29e7137ca4dbb41a1 P 3f367f8f1f824626972b6f16680a369e: Deleting consensus metadata
I20260808 15:48:30.993759 30683 catalog_manager.cc:4841] TS 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045): tablet 1948b634183140f29e7137ca4dbb41a1 (table test-table [id=c72acb4f480c4524a2886c21189677cc]) successfully deleted
I20260808 15:48:30.994218 30681 catalog_manager.cc:4841] TS 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147): tablet 1948b634183140f29e7137ca4dbb41a1 (table test-table [id=c72acb4f480c4524a2886c21189677cc]) successfully deleted
I20260808 15:48:30.998111 31637 ts_tablet_manager.cc:1918] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:48:30.998297 31637 log.cc:1196] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/1948b634183140f29e7137ca4dbb41a1
I20260808 15:48:30.998867 31637 ts_tablet_manager.cc:1939] T 1948b634183140f29e7137ca4dbb41a1 P 78ca76d31d8e40b89806a9cb624bb630: Deleting consensus metadata
I20260808 15:48:30.999908 30683 catalog_manager.cc:4841] TS 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799): tablet 1948b634183140f29e7137ca4dbb41a1 (table test-table [id=c72acb4f480c4524a2886c21189677cc]) successfully deleted
Aug 08 15:48:31 dist-test-slave-twwt krb5kdc[30629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:48:31 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:31.038810 31653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:31.022075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.27.84.254:33643)
0808 15:48:31.022839 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:31.022845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:31.022870 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:48:31.023110 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:31.023115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:31.023198 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:31.023312 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:31.024783 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.025815 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.027396 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.028050 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.028168 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:31.033126 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:31.033132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:31.033135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:31.033177 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:31.036235 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:31.036939 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:31.036943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:31.036945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:31.037082 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:31.037440 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:31.037443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:31.037445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:31.037718 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:31.037899 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:31.038192 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 15:48:31.038382 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":354,"threads_started":1}
I20260808 15:48:31.042587 30706 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:31.043375 30706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:31.051087 30926 tablet_service.cc:1467] Processing CreateTablet for tablet e9bd7556b9a341f5af442240a990e841 (DEFAULT_TABLE table=test-table [id=e0df3465ef1d4c60bfc19ccabd253ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:31.051424 31350 tablet_service.cc:1467] Processing CreateTablet for tablet e9bd7556b9a341f5af442240a990e841 (DEFAULT_TABLE table=test-table [id=e0df3465ef1d4c60bfc19ccabd253ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:31.051617 30926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bd7556b9a341f5af442240a990e841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:31.051724 31144 tablet_service.cc:1467] Processing CreateTablet for tablet e9bd7556b9a341f5af442240a990e841 (DEFAULT_TABLE table=test-table [id=e0df3465ef1d4c60bfc19ccabd253ad6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:31.052003 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bd7556b9a341f5af442240a990e841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:31.052206 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9bd7556b9a341f5af442240a990e841. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:31.056851 31656 tablet_bootstrap.cc:492] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Bootstrap starting.
I20260808 15:48:31.058549 31656 tablet_bootstrap.cc:654] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:31.060791 31659 tablet_bootstrap.cc:492] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Bootstrap starting.
I20260808 15:48:31.063361 31659 tablet_bootstrap.cc:654] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:31.064203 31660 tablet_bootstrap.cc:492] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Bootstrap starting.
I20260808 15:48:31.066282 31660 tablet_bootstrap.cc:654] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:31.066892 31659 tablet_bootstrap.cc:492] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: No bootstrap required, opened a new log
I20260808 15:48:31.067076 31659 ts_tablet_manager.cc:1397] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260808 15:48:31.068346 31659 raft_consensus.cc:348] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.068774 31659 raft_consensus.cc:374] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:31.068908 31659 raft_consensus.cc:729] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f367f8f1f824626972b6f16680a369e, State: Initialized, Role: FOLLOWER
I20260808 15:48:31.069235 31659 consensus_queue.cc:260] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.069978 31659 ts_tablet_manager.cc:1428] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20260808 15:48:31.071740 31656 tablet_bootstrap.cc:492] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: No bootstrap required, opened a new log
I20260808 15:48:31.071914 31656 ts_tablet_manager.cc:1397] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.005s
I20260808 15:48:31.072921 31660 tablet_bootstrap.cc:492] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: No bootstrap required, opened a new log
I20260808 15:48:31.072871 31656 raft_consensus.cc:348] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.073143 31660 ts_tablet_manager.cc:1397] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260808 15:48:31.073172 31656 raft_consensus.cc:374] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:31.073347 31656 raft_consensus.cc:729] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Initialized, Role: FOLLOWER
I20260808 15:48:31.073647 31656 consensus_queue.cc:260] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.074036 31660 raft_consensus.cc:348] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.074283 31660 raft_consensus.cc:374] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:31.074378 31660 raft_consensus.cc:729] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Initialized, Role: FOLLOWER
I20260808 15:48:31.074442 31656 ts_tablet_manager.cc:1428] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 15:48:31.074738 31660 consensus_queue.cc:260] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.075835 31660 ts_tablet_manager.cc:1428] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260808 15:48:31.166059 31602 raft_consensus.cc:482] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:31.166292 31602 raft_consensus.cc:504] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.167140 31602 leader_election.cc:288] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:31.168001 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bd7556b9a341f5af442240a990e841" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e" is_pre_election: true
I20260808 15:48:31.168030 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bd7556b9a341f5af442240a990e841" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca76d31d8e40b89806a9cb624bb630" is_pre_election: true
I20260808 15:48:31.168303 31168 raft_consensus.cc:2454] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 0.
I20260808 15:48:31.168318 31400 raft_consensus.cc:2454] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 0.
I20260808 15:48:31.168908 30858 leader_election.cc:302] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 4c3a7d68ace9422fb626b4e15ebdc9df; no voters:
I20260808 15:48:31.169291 31602 raft_consensus.cc:2781] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:31.169433 31602 raft_consensus.cc:482] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:31.169526 31602 raft_consensus.cc:3037] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:31.172960 31602 raft_consensus.cc:504] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.173731 31602 leader_election.cc:288] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 election: Requested vote from peers 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:31.174423 31168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bd7556b9a341f5af442240a990e841" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78ca76d31d8e40b89806a9cb624bb630"
I20260808 15:48:31.174674 31168 raft_consensus.cc:3037] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:31.174535 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9bd7556b9a341f5af442240a990e841" candidate_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e"
I20260808 15:48:31.174862 31400 raft_consensus.cc:3037] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:31.177196 31400 raft_consensus.cc:2454] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 1.
I20260808 15:48:31.177767 30858 leader_election.cc:302] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 4c3a7d68ace9422fb626b4e15ebdc9df; no voters:
I20260808 15:48:31.177922 31168 raft_consensus.cc:2454] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c3a7d68ace9422fb626b4e15ebdc9df in term 1.
I20260808 15:48:31.178124 31602 raft_consensus.cc:2781] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:31.178341 31602 raft_consensus.cc:686] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 LEADER]: Becoming Leader. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Running, Role: LEADER
I20260808 15:48:31.178669 31602 consensus_queue.cc:237] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.181902 30703 catalog_manager.cc:5495] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193). New cstate: current_term: 1 leader_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } health_report { overall_health: UNKNOWN } } }
I20260808 15:48:31.236377 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:31.230498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.27.84.193:46045)
0808 15:48:31.231250 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:31.231255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:31.231276 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:48:31.231334 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:31.231337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:31.231402 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:31.231490 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:31.232025 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.233098 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.234541 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.235027 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.235125 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:31.235573 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:31.235688 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:31.235756 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:31.235912 (+ 156us) server_negotiation.cc:299] Negotiation successful
0808 15:48:31.235974 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":407,"threads_started":1}
I20260808 15:48:31.250903 31671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:31.245901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:37009 (local address 127.27.84.193:46045)
0808 15:48:31.246081 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:31.246086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:31.246357 (+ 271us) server_negotiation.cc:407] Connection header received
0808 15:48:31.246481 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:31.246484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:31.246544 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:31.246682 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:31.247141 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.247955 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.249287 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.249703 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.249794 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:31.250152 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:31.250245 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:31.250326 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:31.250494 (+ 168us) server_negotiation.cc:299] Negotiation successful
0808 15:48:31.250587 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":56}
I20260808 15:48:31.251087 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:31.245432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:46045 (local address 127.27.84.193:37009)
0808 15:48:31.246245 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:31.246261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:31.246382 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:31.246703 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:31.246706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:31.246715 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:48:31.247001 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:31.247010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.248083 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.248087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:31.249155 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:31.249168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.249853 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.249857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:31.249972 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:31.250010 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:48:31.250371 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:48:31.250382 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:48:31.250458 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 15:48:31.250580 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":2,"thread_start_us":542,"threads_started":1}
I20260808 15:48:31.259320 31400 raft_consensus.cc:1261] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Refusing update from remote peer 4c3a7d68ace9422fb626b4e15ebdc9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:31.259356 31168 raft_consensus.cc:1261] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Refusing update from remote peer 4c3a7d68ace9422fb626b4e15ebdc9df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:31.259881 31602 consensus_queue.cc:1035] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:31.260222 31608 consensus_queue.cc:1035] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [LEADER]: Connected to new peer: Peer: permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:31.316094 30703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:31.316479 30703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:31.320446 30703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Sending DeleteTablet for 3 replicas of tablet e9bd7556b9a341f5af442240a990e841
I20260808 15:48:31.321514 30926 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bd7556b9a341f5af442240a990e841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:31 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260808 15:48:31.321628 31144 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bd7556b9a341f5af442240a990e841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:31 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:46896
I20260808 15:48:31.321851 31636 tablet_replica.cc:330] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: stopping tablet replica
I20260808 15:48:31.321979 31637 tablet_replica.cc:330] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: stopping tablet replica
I20260808 15:48:31.322081 31636 raft_consensus.cc:2229] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:31.322197 31637 raft_consensus.cc:2229] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:31.322420 31637 raft_consensus.cc:2258] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:31.322650 31636 raft_consensus.cc:2258] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:31.323611 31637 ts_tablet_manager.cc:1905] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:31.323660 31636 ts_tablet_manager.cc:1905] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:31.327405 31350 tablet_service.cc:1514] Processing DeleteTablet for tablet e9bd7556b9a341f5af442240a990e841 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:31 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:54180
I20260808 15:48:31.327757 31639 tablet_replica.cc:330] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: stopping tablet replica
I20260808 15:48:31.327981 31639 raft_consensus.cc:2229] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:31.328213 31639 raft_consensus.cc:2258] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:31.329398 31639 ts_tablet_manager.cc:1905] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:31.329519 31637 ts_tablet_manager.cc:1918] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:31.329952 31637 log.cc:1196] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/e9bd7556b9a341f5af442240a990e841
I20260808 15:48:31.330399 31637 ts_tablet_manager.cc:1939] T e9bd7556b9a341f5af442240a990e841 P 78ca76d31d8e40b89806a9cb624bb630: Deleting consensus metadata
I20260808 15:48:31.330477 31636 ts_tablet_manager.cc:1918] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:31.330655 31636 log.cc:1196] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/e9bd7556b9a341f5af442240a990e841
I20260808 15:48:31.331094 31636 ts_tablet_manager.cc:1939] T e9bd7556b9a341f5af442240a990e841 P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting consensus metadata
I20260808 15:48:31.331674 30683 catalog_manager.cc:4841] TS 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799): tablet e9bd7556b9a341f5af442240a990e841 (table test-table [id=e0df3465ef1d4c60bfc19ccabd253ad6]) successfully deleted
I20260808 15:48:31.332626 30683 catalog_manager.cc:4841] TS 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045): tablet e9bd7556b9a341f5af442240a990e841 (table test-table [id=e0df3465ef1d4c60bfc19ccabd253ad6]) successfully deleted
I20260808 15:48:31.335501 31639 ts_tablet_manager.cc:1918] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:48:31.335675 31639 log.cc:1196] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/e9bd7556b9a341f5af442240a990e841
I20260808 15:48:31.336220 31639 ts_tablet_manager.cc:1939] T e9bd7556b9a341f5af442240a990e841 P 3f367f8f1f824626972b6f16680a369e: Deleting consensus metadata
I20260808 15:48:31.337355 30681 catalog_manager.cc:4841] TS 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147): tablet e9bd7556b9a341f5af442240a990e841 (table test-table [id=e0df3465ef1d4c60bfc19ccabd253ad6]) successfully deleted
I20260808 15:48:31.343988 30703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:31.344728 30703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:31.353072 31144 tablet_service.cc:1467] Processing CreateTablet for tablet 4425f85c44464aeda0a8d1d2d5a5b83d (DEFAULT_TABLE table=test-table [id=0a0b40db0324415c85665e1e22093073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:31.353159 31350 tablet_service.cc:1467] Processing CreateTablet for tablet 4425f85c44464aeda0a8d1d2d5a5b83d (DEFAULT_TABLE table=test-table [id=0a0b40db0324415c85665e1e22093073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:31.353662 31350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4425f85c44464aeda0a8d1d2d5a5b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:31.353654 31144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4425f85c44464aeda0a8d1d2d5a5b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:31.358902 31660 tablet_bootstrap.cc:492] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Bootstrap starting.
I20260808 15:48:31.359503 30926 tablet_service.cc:1467] Processing CreateTablet for tablet 4425f85c44464aeda0a8d1d2d5a5b83d (DEFAULT_TABLE table=test-table [id=0a0b40db0324415c85665e1e22093073]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:31.360025 30926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4425f85c44464aeda0a8d1d2d5a5b83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:31.360867 31660 tablet_bootstrap.cc:654] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:31.361999 31659 tablet_bootstrap.cc:492] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Bootstrap starting.
I20260808 15:48:31.364169 31659 tablet_bootstrap.cc:654] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:31.367714 31656 tablet_bootstrap.cc:492] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Bootstrap starting.
I20260808 15:48:31.370124 31659 tablet_bootstrap.cc:492] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: No bootstrap required, opened a new log
I20260808 15:48:31.370210 31656 tablet_bootstrap.cc:654] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:31.370308 31659 ts_tablet_manager.cc:1397] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260808 15:48:31.371429 31659 raft_consensus.cc:348] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.372937 31659 raft_consensus.cc:374] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:31.373063 31659 raft_consensus.cc:729] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f367f8f1f824626972b6f16680a369e, State: Initialized, Role: FOLLOWER
I20260808 15:48:31.373340 31659 consensus_queue.cc:260] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.373991 31659 ts_tablet_manager.cc:1428] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260808 15:48:31.374402 31660 tablet_bootstrap.cc:492] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: No bootstrap required, opened a new log
I20260808 15:48:31.374696 31660 ts_tablet_manager.cc:1397] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.010s
I20260808 15:48:31.375558 31660 raft_consensus.cc:348] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.375859 31660 raft_consensus.cc:374] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:31.375953 31660 raft_consensus.cc:729] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Initialized, Role: FOLLOWER
I20260808 15:48:31.376251 31660 consensus_queue.cc:260] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.376827 31660 ts_tablet_manager.cc:1428] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 15:48:31.378351 31656 tablet_bootstrap.cc:492] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: No bootstrap required, opened a new log
I20260808 15:48:31.378580 31656 ts_tablet_manager.cc:1397] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260808 15:48:31.379472 31656 raft_consensus.cc:348] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.379725 31656 raft_consensus.cc:374] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:31.379854 31656 raft_consensus.cc:729] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3a7d68ace9422fb626b4e15ebdc9df, State: Initialized, Role: FOLLOWER
I20260808 15:48:31.380152 31656 consensus_queue.cc:260] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.380759 31656 ts_tablet_manager.cc:1428] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260808 15:48:31.607097 31665 raft_consensus.cc:482] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:31.607376 31665 raft_consensus.cc:504] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.608454 31665 leader_election.cc:288] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:31.609073 30967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4425f85c44464aeda0a8d1d2d5a5b83d" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" is_pre_election: true
I20260808 15:48:31.609105 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4425f85c44464aeda0a8d1d2d5a5b83d" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e" is_pre_election: true
I20260808 15:48:31.609400 30967 raft_consensus.cc:2454] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 0.
I20260808 15:48:31.609465 31400 raft_consensus.cc:2454] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 0.
I20260808 15:48:31.609988 31066 leader_election.cc:302] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:31.610283 31665 raft_consensus.cc:2781] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:31.610420 31665 raft_consensus.cc:482] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:31.610534 31665 raft_consensus.cc:3037] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:31.612965 31665 raft_consensus.cc:504] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.613549 31665 leader_election.cc:288] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 election: Requested vote from peers 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045), 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147)
I20260808 15:48:31.614157 31400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4425f85c44464aeda0a8d1d2d5a5b83d" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f367f8f1f824626972b6f16680a369e"
I20260808 15:48:31.614190 30967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4425f85c44464aeda0a8d1d2d5a5b83d" candidate_uuid: "78ca76d31d8e40b89806a9cb624bb630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df"
I20260808 15:48:31.614329 31400 raft_consensus.cc:3037] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:31.614423 30967 raft_consensus.cc:3037] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:31.616827 31400 raft_consensus.cc:2454] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 1.
I20260808 15:48:31.617362 31066 leader_election.cc:302] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f367f8f1f824626972b6f16680a369e, 78ca76d31d8e40b89806a9cb624bb630; no voters:
I20260808 15:48:31.617771 31665 raft_consensus.cc:2781] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:31.618242 30967 raft_consensus.cc:2454] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ca76d31d8e40b89806a9cb624bb630 in term 1.
I20260808 15:48:31.618412 31665 raft_consensus.cc:686] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 1 LEADER]: Becoming Leader. State: Replica: 78ca76d31d8e40b89806a9cb624bb630, State: Running, Role: LEADER
I20260808 15:48:31.618896 31665 consensus_queue.cc:237] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } }
I20260808 15:48:31.622409 30703 catalog_manager.cc:5495] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "78ca76d31d8e40b89806a9cb624bb630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ca76d31d8e40b89806a9cb624bb630" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 } health_report { overall_health: UNKNOWN } } }
I20260808 15:48:31.678758 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:31.672745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.27.84.194:37799)
0808 15:48:31.673410 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:31.673414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:31.673434 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:48:31.673487 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:31.673491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:31.673553 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:31.673672 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:31.674135 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.675193 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.676710 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:31.677181 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:31.677330 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:31.677773 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:31.677899 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:31.678007 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:31.678158 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 15:48:31.678242 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":383,"threads_started":1}
I20260808 15:48:31.690021 31400 raft_consensus.cc:1261] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Refusing update from remote peer 78ca76d31d8e40b89806a9cb624bb630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:31.690338 30967 raft_consensus.cc:1261] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Refusing update from remote peer 78ca76d31d8e40b89806a9cb624bb630: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:31.690697 31694 consensus_queue.cc:1035] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f367f8f1f824626972b6f16680a369e" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:31.691190 31665 consensus_queue.cc:1035] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3a7d68ace9422fb626b4e15ebdc9df" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 46045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:48:31.719242 31611 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260808 15:48:33.319687 30703 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:33.320084 30703 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:33.324812 30703 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9056937fb38d4556870efb71ddb73b8a: Sending DeleteTablet for 3 replicas of tablet 4425f85c44464aeda0a8d1d2d5a5b83d
I20260808 15:48:33.325680 30926 tablet_service.cc:1514] Processing DeleteTablet for tablet 4425f85c44464aeda0a8d1d2d5a5b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:33 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:47190
I20260808 15:48:33.325801 31144 tablet_service.cc:1514] Processing DeleteTablet for tablet 4425f85c44464aeda0a8d1d2d5a5b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:33 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:46896
I20260808 15:48:33.326045 31350 tablet_service.cc:1514] Processing DeleteTablet for tablet 4425f85c44464aeda0a8d1d2d5a5b83d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:33 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:54180
I20260808 15:48:33.328879 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30828
I20260808 15:48:33.342061 31735 tablet_replica.cc:330] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: stopping tablet replica
I20260808 15:48:33.342304 31735 raft_consensus.cc:2229] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:33.343212 31735 raft_consensus.cc:2258] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:33.343806 31736 tablet_replica.cc:330] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: stopping tablet replica
I20260808 15:48:33.344015 31736 raft_consensus.cc:2229] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:33.344370 31736 raft_consensus.cc:2258] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:33.344482 31735 ts_tablet_manager.cc:1905] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:33.345293 31736 ts_tablet_manager.cc:1905] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:33.347925 31737 tablet_replica.cc:330] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: stopping tablet replica
I20260808 15:48:33.348134 31737 raft_consensus.cc:2229] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:33.348420 31737 raft_consensus.cc:2258] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:33.351610 31735 ts_tablet_manager.cc:1918] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:48:33.351769 31735 log.cc:1196] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/4425f85c44464aeda0a8d1d2d5a5b83d
I20260808 15:48:33.352066 31736 ts_tablet_manager.cc:1918] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:48:33.352240 31736 log.cc:1196] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/4425f85c44464aeda0a8d1d2d5a5b83d
I20260808 15:48:33.352288 31735 ts_tablet_manager.cc:1939] T 4425f85c44464aeda0a8d1d2d5a5b83d P 4c3a7d68ace9422fb626b4e15ebdc9df: Deleting consensus metadata
I20260808 15:48:33.352777 31736 ts_tablet_manager.cc:1939] T 4425f85c44464aeda0a8d1d2d5a5b83d P 78ca76d31d8e40b89806a9cb624bb630: Deleting consensus metadata
I20260808 15:48:33.352938 31737 ts_tablet_manager.cc:1905] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:33.353974 30683 catalog_manager.cc:4841] TS 4c3a7d68ace9422fb626b4e15ebdc9df (127.27.84.193:46045): tablet 4425f85c44464aeda0a8d1d2d5a5b83d (table test-table [id=0a0b40db0324415c85665e1e22093073]) successfully deleted
I20260808 15:48:33.354514 30683 catalog_manager.cc:4841] TS 78ca76d31d8e40b89806a9cb624bb630 (127.27.84.194:37799): tablet 4425f85c44464aeda0a8d1d2d5a5b83d (table test-table [id=0a0b40db0324415c85665e1e22093073]) successfully deleted
Aug 08 15:48:33 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:33.359998 31737 ts_tablet_manager.cc:1918] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260808 15:48:33.360143 31737 log.cc:1196] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/4425f85c44464aeda0a8d1d2d5a5b83d
I20260808 15:48:33.360631 31737 ts_tablet_manager.cc:1939] T 4425f85c44464aeda0a8d1d2d5a5b83d P 3f367f8f1f824626972b6f16680a369e: Deleting consensus metadata
I20260808 15:48:33.361835 30681 catalog_manager.cc:4841] TS 3f367f8f1f824626972b6f16680a369e (127.27.84.195:37147): tablet 4425f85c44464aeda0a8d1d2d5a5b83d (table test-table [id=0a0b40db0324415c85665e1e22093073]) successfully deleted
I20260808 15:48:33.366652 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:33.330106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.27.84.193:46045)
0808 15:48:33.347056 (+ 16950us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:33.347064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:33.347152 (+ 88us) server_negotiation.cc:407] Connection header received
0808 15:48:33.347220 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:33.347225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:33.347348 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:33.350751 (+ 3403us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:33.350962 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:33.352102 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:33.354219 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:33.354980 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:33.355148 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:33.359505 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:33.359521 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:33.359539 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:33.359572 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:33.363393 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:33.364145 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:33.364152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:33.364154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:33.364302 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:33.364733 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:33.364738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:33.364864 (+ 126us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:33.365301 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:33.365647 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:33.365987 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 15:48:33.366226 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16813,"thread_start_us":16655,"threads_started":1}
W20260808 15:48:33.367435 31030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52528
W20260808 15:48:33.368094 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30828: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:33.368396 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30828
I20260808 15:48:33.408975 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31044
Aug 08 15:48:33 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:33.425566 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:33.409716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.27.84.194:37799)
0808 15:48:33.410291 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:33.410295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:33.410312 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:48:33.410382 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:33.410386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:33.410445 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:33.410539 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:33.411877 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:33.412738 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:33.414132 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:33.414674 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:33.414799 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:33.419247 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:33.419264 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:33.419281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:33.419323 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:33.422780 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:33.423486 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:33.423494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:33.423497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:33.423651 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:33.424120 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:33.424124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:33.424126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:33.424530 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:33.424766 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:33.425102 (+ 336us) server_negotiation.cc:299] Negotiation successful
0808 15:48:33.425280 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":333,"threads_started":1}
W20260808 15:48:33.426362 31241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37934
W20260808 15:48:33.427140 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31044: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:33.427363 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 31044
I20260808 15:48:33.462484 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31256
Aug 08 15:48:33 dist-test-slave-twwt krb5kdc[30629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:33.480535 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:33.463389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.27.84.195:37147)
0808 15:48:33.463987 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:33.463991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:33.464008 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:48:33.464067 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:33.464071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:33.464138 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:33.464234 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:33.465536 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:33.466305 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:33.468274 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:33.468712 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:33.468853 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:33.474245 (+ 5392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:33.474262 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:33.474279 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:33.474325 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:33.477438 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:33.478167 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:33.478176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:33.478178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:33.478325 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:33.478889 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:33.478896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:33.478899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:33.479278 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:33.479591 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:33.479930 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:48:33.480179 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":304,"threads_started":1}
W20260808 15:48:33.481261 31464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37234
W20260808 15:48:33.481964 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31256: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:33.482159 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 31256
I20260808 15:48:33.515492 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30652
I20260808 15:48:33.516748 30771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:33.638077 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30652
2026-08-08T15:48:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9220 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:33 dist-test-slave-twwt krb5kdc[31755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:33 dist-test-slave-twwt krb5kdc[31755](info): set up 2 sockets
Aug 08 15:48:33 dist-test-slave-twwt krb5kdc[31755](info): commencing operation
krb5kdc: starting...
W20260808 15:48:34.983774 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:35 dist-test-slave-twwt krb5kdc[31755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:35.226881 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36213
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:35.569161 31775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:35.569527 31775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:35.569701 31775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:35.576970 31775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:35.577134 31775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:35.577248 31775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:35.577358 31775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:35.577448 31775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:35.587358 31775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:35.588810 31775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:35.590222 31775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:35.603175 31783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:35.603262 31784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:35.603592 31787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:35.604131 31775 server_base.cc:1034] running on GCE node
I20260808 15:48:35.605044 31775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:35.606760 31775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:35.607995 31775 hybrid_clock.cc:648] HybridClock initialized: now 1786204115607947 us; error 43 us; skew 500 ppm
I20260808 15:48:35.608412 31775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:35 dist-test-slave-twwt krb5kdc[31755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204115, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:35.612155 31775 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:35.614012 31775 webserver.cc:460] Webserver started at http://127.27.84.254:38595/ using document root <none> and password file <none>
I20260808 15:48:35.614917 31775 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:35.615067 31775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:35.615451 31775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:35.618957 31775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "3d3f33a9d848432d8b5a450e1cf4f969"
format_stamp: "Formatted at 2026-08-08 15:48:35 on dist-test-slave-twwt"
server_key: "ab063d7889e6d640b51ed7c1e4798c7f"
server_key_iv: "f28f4c56479bf2fd680666b83df6ede3"
server_key_version: "encryptionkey@0"
I20260808 15:48:35.619980 31775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "3d3f33a9d848432d8b5a450e1cf4f969"
format_stamp: "Formatted at 2026-08-08 15:48:35 on dist-test-slave-twwt"
server_key: "ab063d7889e6d640b51ed7c1e4798c7f"
server_key_iv: "f28f4c56479bf2fd680666b83df6ede3"
server_key_version: "encryptionkey@0"
I20260808 15:48:35.626084 31775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 15:48:35.630236 31798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:35.631915 31775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:35.632133 31775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "3d3f33a9d848432d8b5a450e1cf4f969"
format_stamp: "Formatted at 2026-08-08 15:48:35 on dist-test-slave-twwt"
server_key: "ab063d7889e6d640b51ed7c1e4798c7f"
server_key_iv: "f28f4c56479bf2fd680666b83df6ede3"
server_key_version: "encryptionkey@0"
I20260808 15:48:35.632360 31775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:35.650672 31775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:35.655826 31775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:35.656183 31775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:35.676909 31775 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:36213
I20260808 15:48:35.676934 31883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:36213 every 8 connection(s)
I20260808 15:48:35.678817 31775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:35.681272 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 31775
I20260808 15:48:35.681520 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:35.682231 27987 external_mini_cluster.cc:1428] Setting key 812c1752a3ccfc6a9f34fdebce53a655
I20260808 15:48:35.683430 31884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:35.696655 31884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969: Bootstrap starting.
Aug 08 15:48:35 dist-test-slave-twwt krb5kdc[31755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:35.700417 31884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:35.701588 31884 log.cc:824] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:35.704794 31884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969: No bootstrap required, opened a new log
I20260808 15:48:35.705761 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:35.684515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.27.84.254:36213)
0808 15:48:35.685355 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:35.685367 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:35.685410 (+ 43us) server_negotiation.cc:407] Connection header received
0808 15:48:35.687184 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:35.687209 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:35.687633 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:35.688083 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:35.689640 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:35.691097 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:35.692403 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:35.693152 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:35.693391 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:35.697974 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:35.697983 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:35.697999 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:35.698043 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:35.701389 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:35.702229 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:35.702238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:35.702246 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:35.702434 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:35.702880 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:35.702887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:35.702895 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:35.703440 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:35.703767 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:35.704227 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 15:48:35.704676 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":372,"threads_started":1}
I20260808 15:48:35.712882 31884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36213 } }
I20260808 15:48:35.713379 31884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:35.713496 31884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3f33a9d848432d8b5a450e1cf4f969, State: Initialized, Role: FOLLOWER
I20260808 15:48:35.714114 31884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36213 } }
I20260808 15:48:35.714432 31884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:35.714591 31884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:35.714879 31884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:35.717281 31884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36213 } }
I20260808 15:48:35.717829 31884 leader_election.cc:302] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d3f33a9d848432d8b5a450e1cf4f969; no voters:
I20260808 15:48:35.718461 31884 leader_election.cc:288] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:35.718636 31889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:35.719447 31889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [term 1 LEADER]: Becoming Leader. State: Replica: 3d3f33a9d848432d8b5a450e1cf4f969, State: Running, Role: LEADER
I20260808 15:48:35.719969 31889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36213 } }
I20260808 15:48:35.720927 31884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:35.723232 31891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d3f33a9d848432d8b5a450e1cf4f969. Latest consensus state: current_term: 1 leader_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36213 } } }
I20260808 15:48:35.723698 31891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:35.724542 31890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f33a9d848432d8b5a450e1cf4f969" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36213 } } }
I20260808 15:48:35.724910 31890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:35.732178 31895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:35.737494 31895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:35.747391 31895 catalog_manager.cc:1347] Generated new cluster ID: e29162a075ce414dbc702bb12d1f6c9d
I20260808 15:48:35.747560 31895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:35.771334 31895 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:35.772861 31895 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:35.780732 31895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d3f33a9d848432d8b5a450e1cf4f969: Generated new TSK 0
I20260808 15:48:35.781694 31895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:35.842456 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36213
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:36.202786 31920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:36.203142 31920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:36.203320 31920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:36.210971 31920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:36.211135 31920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:36.211331 31920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:36.222018 31920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:36.223568 31920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:36.225291 31920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:36.240618 31932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:36.240769 31926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:36.241242 31927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:36.241843 31920 server_base.cc:1034] running on GCE node
I20260808 15:48:36.242694 31920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:36.243846 31920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:36.245081 31920 hybrid_clock.cc:648] HybridClock initialized: now 1786204116245008 us; error 69 us; skew 500 ppm
I20260808 15:48:36.245486 31920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:36 dist-test-slave-twwt krb5kdc[31755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204116, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:36.249361 31920 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:36.251137 31920 webserver.cc:460] Webserver started at http://127.27.84.193:38449/ using document root <none> and password file <none>
I20260808 15:48:36.251896 31920 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:36.252019 31920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:36.252331 31920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:36.255750 31920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "2b66729137dd45ebbae63f50df4ddf17"
format_stamp: "Formatted at 2026-08-08 15:48:36 on dist-test-slave-twwt"
server_key: "e2af7cd773558a17a6565475ecd1ce24"
server_key_iv: "7af665fbc08160b0f9da332e160b532a"
server_key_version: "encryptionkey@0"
I20260808 15:48:36.256624 31920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "2b66729137dd45ebbae63f50df4ddf17"
format_stamp: "Formatted at 2026-08-08 15:48:36 on dist-test-slave-twwt"
server_key: "e2af7cd773558a17a6565475ecd1ce24"
server_key_iv: "7af665fbc08160b0f9da332e160b532a"
server_key_version: "encryptionkey@0"
I20260808 15:48:36.262566 31920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:48:36.266744 31945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:36.268361 31920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:36.268518 31920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "2b66729137dd45ebbae63f50df4ddf17"
format_stamp: "Formatted at 2026-08-08 15:48:36 on dist-test-slave-twwt"
server_key: "e2af7cd773558a17a6565475ecd1ce24"
server_key_iv: "7af665fbc08160b0f9da332e160b532a"
server_key_version: "encryptionkey@0"
I20260808 15:48:36.268708 31920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:36.291913 31920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:36.297446 31920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:36.297837 31920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:36.299055 31920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:36.300580 31920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:36.300666 31920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:36.300871 31920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:36.300936 31920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:36.339182 31920 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:33919
I20260808 15:48:36.339231 32125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:33919 every 8 connection(s)
I20260808 15:48:36.340687 31920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:36.348960 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 31920
I20260808 15:48:36.349246 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:36.350070 27987 external_mini_cluster.cc:1428] Setting key c88556fd597fa03d8c7c7e5fc6fbe40e
Aug 08 15:48:36 dist-test-slave-twwt krb5kdc[31755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204116, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:36.364782 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:36.343576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:57105 (local address 127.27.84.254:36213)
0808 15:48:36.344057 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:36.344062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:36.345175 (+ 1113us) server_negotiation.cc:407] Connection header received
0808 15:48:36.347272 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:36.347278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:36.347387 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:36.347516 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:36.349229 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.350401 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.352624 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.353293 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.353445 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:36.357395 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:36.357404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:36.357411 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:36.357460 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:36.361046 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:36.362164 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:36.362168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:36.362171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:36.362302 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:36.362893 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:36.362897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:36.362899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:36.363176 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:36.363307 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:36.364333 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0808 15:48:36.364514 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260808 15:48:36.366552 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:36.344166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36213 (local address 127.27.84.193:57105)
0808 15:48:36.344982 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:36.345037 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:36.346969 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:36.347798 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:36.347808 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:36.348395 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:36.349011 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:36.349030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.350571 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.350577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:36.352468 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:36.352483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.353541 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.353547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:36.353763 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:36.354564 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:36.354596 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:36.357114 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:36.361213 (+ 4099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:36.361228 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:36.361250 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:36.362010 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:36.362426 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:36.362434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:36.362441 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:36.362757 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:36.363335 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:36.363348 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:36.363851 (+ 503us) client_negotiation.cc:769] Sending connection context
0808 15:48:36.364188 (+ 337us) client_negotiation.cc:240] Negotiation successful
0808 15:48:36.364848 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":292,"threads_started":1}
I20260808 15:48:36.368295 32127 heartbeater.cc:347] Connected to a master server at 127.27.84.254:36213
I20260808 15:48:36.368626 32127 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:36.369405 32127 heartbeater.cc:511] Master 127.27.84.254:36213 requested a full tablet report, sending...
I20260808 15:48:36.371721 31817 ts_manager.cc:194] Registered new tserver with Master: 2b66729137dd45ebbae63f50df4ddf17 (127.27.84.193:33919)
I20260808 15:48:36.373283 31817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:57105
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:36.407297 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36213
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:36.754488 32136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:36.754873 32136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:36.755059 32136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:36.762476 32136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:36.762697 32136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:36.762913 32136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:36.773370 32136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:36.774891 32136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:36.776350 32136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:36.790644 32145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:36.790593 32146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:36.790545 32149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:36.791739 32136 server_base.cc:1034] running on GCE node
I20260808 15:48:36.792388 32136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:36.793422 32136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:36.794699 32136 hybrid_clock.cc:648] HybridClock initialized: now 1786204116794639 us; error 53 us; skew 500 ppm
I20260808 15:48:36.795089 32136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:36 dist-test-slave-twwt krb5kdc[31755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204116, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:36.798431 32136 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:36.800053 32136 webserver.cc:460] Webserver started at http://127.27.84.194:38997/ using document root <none> and password file <none>
I20260808 15:48:36.800714 32136 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:36.800823 32136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:36.801086 32136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:36.804143 32136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "0b62456c31b94d2ea02ab01f6255602a"
format_stamp: "Formatted at 2026-08-08 15:48:36 on dist-test-slave-twwt"
server_key: "18fdda30744979196590d12f981dc813"
server_key_iv: "1c0ad7957e7060294483847aa7322462"
server_key_version: "encryptionkey@0"
I20260808 15:48:36.804927 32136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "0b62456c31b94d2ea02ab01f6255602a"
format_stamp: "Formatted at 2026-08-08 15:48:36 on dist-test-slave-twwt"
server_key: "18fdda30744979196590d12f981dc813"
server_key_iv: "1c0ad7957e7060294483847aa7322462"
server_key_version: "encryptionkey@0"
I20260808 15:48:36.810511 32136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:48:36.814390 32156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:36.815986 32136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:36.816143 32136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "0b62456c31b94d2ea02ab01f6255602a"
format_stamp: "Formatted at 2026-08-08 15:48:36 on dist-test-slave-twwt"
server_key: "18fdda30744979196590d12f981dc813"
server_key_iv: "1c0ad7957e7060294483847aa7322462"
server_key_version: "encryptionkey@0"
I20260808 15:48:36.816330 32136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:36.841432 32136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:36.846154 32136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:36.846434 32136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:36.847499 32136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:36.848901 32136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:36.848976 32136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:36.849062 32136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:36.849107 32136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:36.884594 32136 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:33133
I20260808 15:48:36.884687 32331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:33133 every 8 connection(s)
I20260808 15:48:36.885943 32136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:36.891054 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 32136
I20260808 15:48:36.891247 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:36.891710 27987 external_mini_cluster.cc:1428] Setting key 32d7f01a5e6353334fbafb05b237e239
Aug 08 15:48:36 dist-test-slave-twwt krb5kdc[31755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204116, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:36.910331 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:36.888685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:48071 (local address 127.27.84.254:36213)
0808 15:48:36.889183 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:36.889190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:36.890162 (+ 972us) server_negotiation.cc:407] Connection header received
0808 15:48:36.892255 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:36.892261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:36.892356 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:36.892506 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:36.894493 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.895736 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.897497 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.898241 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.898397 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:36.902975 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:36.902985 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:36.902991 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:36.903039 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:36.906738 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:36.907594 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:36.907603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:36.907610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:36.907781 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:36.908223 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:36.908231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:36.908238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:36.908654 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:36.908823 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:36.909568 (+ 745us) server_negotiation.cc:299] Negotiation successful
0808 15:48:36.909835 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
I20260808 15:48:36.911372 32335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:36.889221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36213 (local address 127.27.84.194:48071)
0808 15:48:36.889980 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:36.890031 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:36.891970 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:36.892804 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:36.892817 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:36.893638 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:36.894307 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:36.894326 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.895879 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.895882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:36.897332 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:36.897350 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:36.898477 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:36.898485 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:36.898801 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:36.899615 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:36.899648 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:36.902702 (+ 3054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:36.906919 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:36.906927 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:36.906942 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:36.907458 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:36.907900 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:36.907903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:36.907905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:36.908109 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:36.908801 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:36.908807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:36.909151 (+ 344us) client_negotiation.cc:769] Sending connection context
0808 15:48:36.909418 (+ 267us) client_negotiation.cc:240] Negotiation successful
0808 15:48:36.909883 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":281,"threads_started":1}
I20260808 15:48:36.913082 32332 heartbeater.cc:347] Connected to a master server at 127.27.84.254:36213
I20260808 15:48:36.913460 32332 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:36.914497 32332 heartbeater.cc:511] Master 127.27.84.254:36213 requested a full tablet report, sending...
I20260808 15:48:36.916436 31817 ts_manager.cc:194] Registered new tserver with Master: 0b62456c31b94d2ea02ab01f6255602a (127.27.84.194:33133)
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
I20260808 15:48:36.917735 31817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:48071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:36.950700 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36213
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:37.300194 32343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:37.300525 32343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:37.300698 32343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:37.308207 32343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:37.308333 32343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:37.308519 32343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:37.319002 32343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:36213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:37.320438 32343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:37.322176 32343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:37.336354 32349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:37.336326 32348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:37.336560 32353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:37.337467 32343 server_base.cc:1034] running on GCE node
I20260808 15:48:37.338236 32343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:37.339465 32343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:37.340746 32343 hybrid_clock.cc:648] HybridClock initialized: now 1786204117340685 us; error 60 us; skew 500 ppm
I20260808 15:48:37.341112 32343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:37 dist-test-slave-twwt krb5kdc[31755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:37.345104 32343 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:37.346762 32343 webserver.cc:460] Webserver started at http://127.27.84.195:45897/ using document root <none> and password file <none>
I20260808 15:48:37.347532 32343 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:37.347673 32343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:37.348028 32343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:37.351230 32343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "1f4d03b3dde54a30b907621882727874"
format_stamp: "Formatted at 2026-08-08 15:48:37 on dist-test-slave-twwt"
server_key: "a21e2b1468c2c1921d60ca38c014c31f"
server_key_iv: "112d3f125bbfdf7903a8d3a073e30c7e"
server_key_version: "encryptionkey@0"
I20260808 15:48:37.352108 32343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "1f4d03b3dde54a30b907621882727874"
format_stamp: "Formatted at 2026-08-08 15:48:37 on dist-test-slave-twwt"
server_key: "a21e2b1468c2c1921d60ca38c014c31f"
server_key_iv: "112d3f125bbfdf7903a8d3a073e30c7e"
server_key_version: "encryptionkey@0"
I20260808 15:48:37.357960 32343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:48:37.361827 32362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:37.363335 32343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:48:37.363552 32343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "1f4d03b3dde54a30b907621882727874"
format_stamp: "Formatted at 2026-08-08 15:48:37 on dist-test-slave-twwt"
server_key: "a21e2b1468c2c1921d60ca38c014c31f"
server_key_iv: "112d3f125bbfdf7903a8d3a073e30c7e"
server_key_version: "encryptionkey@0"
I20260808 15:48:37.363785 32343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:37.376580 32127 heartbeater.cc:503] Master 127.27.84.254:36213 was elected leader, sending a full tablet report...
I20260808 15:48:37.390902 32343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:37.395633 32343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:37.395943 32343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:37.396956 32343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:37.398332 32343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:37.398439 32343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:37.398591 32343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:37.398699 32343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 15:48:37.433467 32343 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:43201
I20260808 15:48:37.433519 32540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:43201 every 8 connection(s)
I20260808 15:48:37.434876 32343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:37.445053 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 32343
I20260808 15:48:37.445247 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:37.445679 27987 external_mini_cluster.cc:1428] Setting key 8834013e42e8ebb8374ae012ea3ee935
Aug 08 15:48:37 dist-test-slave-twwt krb5kdc[31755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:37.455185 32543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:37.437562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:55811 (local address 127.27.84.254:36213)
0808 15:48:37.438047 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:37.438053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:37.439159 (+ 1106us) server_negotiation.cc:407] Connection header received
0808 15:48:37.440983 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:37.440987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:37.441054 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:37.441150 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:37.442702 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:37.443491 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:37.445028 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:37.445554 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:37.445693 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:37.449400 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:37.449406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:37.449409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:37.449443 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:37.452065 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:37.452805 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:37.452813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:37.452816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:37.452950 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:37.453386 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:37.453392 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:37.453397 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:37.453673 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:37.453839 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:37.454514 (+ 675us) server_negotiation.cc:299] Negotiation successful
0808 15:48:37.454766 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":214,"threads_started":1}
I20260808 15:48:37.456143 32544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:37.438189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36213 (local address 127.27.84.195:55811)
0808 15:48:37.438992 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:37.439041 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:37.440720 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:37.441370 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:37.441378 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:37.441917 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:37.442501 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:37.442519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:37.443630 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:37.443634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:37.444868 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:37.444885 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:37.445799 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:37.445807 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:37.446047 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:37.446802 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:37.446825 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:37.449186 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:37.452213 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:37.452222 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:37.452238 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:37.452671 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:37.453067 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:37.453072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:37.453075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:37.453271 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:37.453808 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:37.453816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:37.454151 (+ 335us) client_negotiation.cc:769] Sending connection context
0808 15:48:37.454419 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 15:48:37.455022 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":327,"threads_started":1}
I20260808 15:48:37.457525 32541 heartbeater.cc:347] Connected to a master server at 127.27.84.254:36213
I20260808 15:48:37.457852 32541 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:37.458577 32541 heartbeater.cc:511] Master 127.27.84.254:36213 requested a full tablet report, sending...
I20260808 15:48:37.460398 31816 ts_manager.cc:194] Registered new tserver with Master: 1f4d03b3dde54a30b907621882727874 (127.27.84.195:43201)
I20260808 15:48:37.461544 31816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:55811
I20260808 15:48:37.461571 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:48:37.502053 32543 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:37.499372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.27.84.254:36213)
0808 15:48:37.499611 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:37.499620 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:37.500059 (+ 439us) server_negotiation.cc:407] Connection header received
0808 15:48:37.500213 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:37.500217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:37.500293 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:37.500385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:37.501710 (+ 1325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44332: BlockingRecv error: recv got EOF from 127.0.0.1:44332 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:48:37.505170 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31920
W20260808 15:48:37.508188 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31920: Not authorized: Client connection negotiation failed: client connection to 127.27.84.193:33919: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:48:37.508396 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 31920
I20260808 15:48:37.535835 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 32136
W20260808 15:48:37.538779 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 32136: Not authorized: Client connection negotiation failed: client connection to 127.27.84.194:33133: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:48:37.539000 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 32136
I20260808 15:48:37.563148 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 32343
W20260808 15:48:37.566483 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 32343: Not authorized: Client connection negotiation failed: client connection to 127.27.84.195:43201: server requires authentication, but client does not have Kerberos credentials available
I20260808 15:48:37.566700 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 32343
I20260808 15:48:37.595695 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31775
I20260808 15:48:37.597164 31873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:37.703547 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 31775
2026-08-08T15:48:37Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4053 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:37 dist-test-slave-twwt krb5kdc[32567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:37 dist-test-slave-twwt krb5kdc[32567](info): set up 2 sockets
Aug 08 15:48:37 dist-test-slave-twwt krb5kdc[32567](info): commencing operation
krb5kdc: starting...
W20260808 15:48:39.038357 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:39 dist-test-slave-twwt krb5kdc[32567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:39.283156 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43829
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:39.624291 32586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:39.624585 32586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:39.624753 32586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:39.631946 32586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:39.632054 32586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:39.632166 32586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:39.632247 32586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:39.632318 32586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:39.642222 32586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:39.643641 32586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:39.645054 32586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:39.657827 32594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:39.657922 32596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:39.657876 32603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:39.658906 32586 server_base.cc:1034] running on GCE node
I20260808 15:48:39.659687 32586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:39.661123 32586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:39.662333 32586 hybrid_clock.cc:648] HybridClock initialized: now 1786204119662285 us; error 48 us; skew 500 ppm
I20260808 15:48:39.662803 32586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:39 dist-test-slave-twwt krb5kdc[32567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204119, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:39.670662 32586 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:39.672267 32586 webserver.cc:460] Webserver started at http://127.27.84.254:42515/ using document root <none> and password file <none>
I20260808 15:48:39.673029 32586 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:39.673183 32586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:39.673513 32586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:39.676697 32586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "a9243266d3fa4967b2bda5189e0c10cd"
format_stamp: "Formatted at 2026-08-08 15:48:39 on dist-test-slave-twwt"
server_key: "9ddb7d0709fcf2adda6348d370106f0e"
server_key_iv: "3debed185ab7be7abe4151b2e56fbda6"
server_key_version: "encryptionkey@0"
I20260808 15:48:39.677517 32586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "a9243266d3fa4967b2bda5189e0c10cd"
format_stamp: "Formatted at 2026-08-08 15:48:39 on dist-test-slave-twwt"
server_key: "9ddb7d0709fcf2adda6348d370106f0e"
server_key_iv: "3debed185ab7be7abe4151b2e56fbda6"
server_key_version: "encryptionkey@0"
I20260808 15:48:39.682832 32586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:48:39.686530 32617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:39.687983 32586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:39.688182 32586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "a9243266d3fa4967b2bda5189e0c10cd"
format_stamp: "Formatted at 2026-08-08 15:48:39 on dist-test-slave-twwt"
server_key: "9ddb7d0709fcf2adda6348d370106f0e"
server_key_iv: "3debed185ab7be7abe4151b2e56fbda6"
server_key_version: "encryptionkey@0"
I20260808 15:48:39.688385 32586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:39.701850 32586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:39.706355 32586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:39.706746 32586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:39.726833 32586 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43829
I20260808 15:48:39.726879 32701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43829 every 8 connection(s)
I20260808 15:48:39.728371 32586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:39.732554 32702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:39.737434 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 32586
I20260808 15:48:39.737689 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:39.738548 27987 external_mini_cluster.cc:1428] Setting key b7f1572d23d6d887f04962f95a3a4524
I20260808 15:48:39.745123 32702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd: Bootstrap starting.
I20260808 15:48:39.749051 32702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:39.750275 32702 log.cc:824] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:39.753551 32702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd: No bootstrap required, opened a new log
Aug 08 15:48:39 dist-test-slave-twwt krb5kdc[32567](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:39.763109 32702 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9243266d3fa4967b2bda5189e0c10cd" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43829 } }
I20260808 15:48:39.763609 32702 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:39.763643 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:39.740591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.27.84.254:43829)
0808 15:48:39.741516 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:39.741527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:39.741571 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:48:39.743513 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:39.743539 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:39.743943 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:39.744296 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:39.746254 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:39.747666 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:39.749489 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:39.750222 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:39.750392 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:39.755055 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:39.755063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:39.755080 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:39.755117 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:39.758584 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:39.759625 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:39.759632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:39.759640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:39.759887 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:39.760325 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:39.760328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:39.760330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:39.761000 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:39.761251 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:39.761861 (+ 610us) server_negotiation.cc:299] Negotiation successful
0808 15:48:39.762366 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":442,"threads_started":1}
I20260808 15:48:39.763748 32702 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9243266d3fa4967b2bda5189e0c10cd, State: Initialized, Role: FOLLOWER
I20260808 15:48:39.764406 32702 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9243266d3fa4967b2bda5189e0c10cd" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43829 } }
I20260808 15:48:39.764688 32702 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:39.764886 32702 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:39.765103 32702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:39.768561 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:48:39.768558 32702 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9243266d3fa4967b2bda5189e0c10cd" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43829 } }
I20260808 15:48:39.769160 32702 leader_election.cc:302] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9243266d3fa4967b2bda5189e0c10cd; no voters:
I20260808 15:48:39.769893 32702 leader_election.cc:288] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:39.770022 32713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:39.772389 32702 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:39.776943 32713 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [term 1 LEADER]: Becoming Leader. State: Replica: a9243266d3fa4967b2bda5189e0c10cd, State: Running, Role: LEADER
I20260808 15:48:39.777683 32713 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9243266d3fa4967b2bda5189e0c10cd" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43829 } }
I20260808 15:48:39.781157 32714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9243266d3fa4967b2bda5189e0c10cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9243266d3fa4967b2bda5189e0c10cd" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43829 } } }
I20260808 15:48:39.781695 32714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:39.782387 32716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9243266d3fa4967b2bda5189e0c10cd. Latest consensus state: current_term: 1 leader_uuid: "a9243266d3fa4967b2bda5189e0c10cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9243266d3fa4967b2bda5189e0c10cd" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43829 } } }
I20260808 15:48:39.782727 32716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd [sys.catalog]: This master's current role is: LEADER
W20260808 15:48:39.795516 32737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:48:39.795662 32737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:48:39.795825 32733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:39.797497 32733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:39.806172 32733 catalog_manager.cc:1347] Generated new cluster ID: d92039c2acc64779b756bb3c65a76b21
I20260808 15:48:39.806291 32733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:39.818213 32733 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:39.819609 32733 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:39.829849 32733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9243266d3fa4967b2bda5189e0c10cd: Generated new TSK 0
I20260808 15:48:39.831010 32733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:48:39.833981 32711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:39.824748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.27.84.254:43829)
0808 15:48:39.824938 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:39.824945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:39.825563 (+ 618us) server_negotiation.cc:407] Connection header received
0808 15:48:39.825761 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:39.825767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:39.825843 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:39.825955 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:39.827321 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:39.828155 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:39.830061 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:39.830729 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:39.830877 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:39.832203 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:39.832210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:39.832219 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:39.833113 (+ 894us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:39.833193 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:39.833231 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:39.833353 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:39.833516 (+ 163us) server_negotiation.cc:299] Negotiation successful
0808 15:48:39.833617 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:48:39.842518 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 32586
I20260808 15:48:39.844667 32695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:39.937029 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 32586
2026-08-08T15:48:39Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2237 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:40 dist-test-slave-twwt krb5kdc[32756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:40 dist-test-slave-twwt krb5kdc[32756](info): set up 2 sockets
Aug 08 15:48:40 dist-test-slave-twwt krb5kdc[32756](info): commencing operation
krb5kdc: starting...
W20260808 15:48:41.273849 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.284s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:41 dist-test-slave-twwt krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:41.518784 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34021
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:34021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:41.860677 310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:41.860991 310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:41.861083 310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:41.868387 310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:41.868526 310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:41.868575 310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:41.868636 310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:41.868683 310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:41.878898 310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:34021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:41.880293 310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:41.881732 310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:41.894199 321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:41.894204 323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:41.894150 326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:41.895236 310 server_base.cc:1034] running on GCE node
I20260808 15:48:41.895987 310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:41.897354 310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:41.898571 310 hybrid_clock.cc:648] HybridClock initialized: now 1786204121898532 us; error 33 us; skew 500 ppm
I20260808 15:48:41.899022 310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:41 dist-test-slave-twwt krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:41.902379 310 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:41.904145 310 webserver.cc:460] Webserver started at http://127.27.84.254:46123/ using document root <none> and password file <none>
I20260808 15:48:41.904889 310 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:41.905030 310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:41.905371 310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:41.908603 310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "8618db3375ee427e8c0874f634316a17"
format_stamp: "Formatted at 2026-08-08 15:48:41 on dist-test-slave-twwt"
server_key: "a3ef836f2f0335355c2ba18e246e89e2"
server_key_iv: "4996f20ecc5cdb93fbba891dd6c23a2e"
server_key_version: "encryptionkey@0"
I20260808 15:48:41.909426 310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "8618db3375ee427e8c0874f634316a17"
format_stamp: "Formatted at 2026-08-08 15:48:41 on dist-test-slave-twwt"
server_key: "a3ef836f2f0335355c2ba18e246e89e2"
server_key_iv: "4996f20ecc5cdb93fbba891dd6c23a2e"
server_key_version: "encryptionkey@0"
I20260808 15:48:41.914981 310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:48:41.918510 340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:41.919930 310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:41.920123 310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "8618db3375ee427e8c0874f634316a17"
format_stamp: "Formatted at 2026-08-08 15:48:41 on dist-test-slave-twwt"
server_key: "a3ef836f2f0335355c2ba18e246e89e2"
server_key_iv: "4996f20ecc5cdb93fbba891dd6c23a2e"
server_key_version: "encryptionkey@0"
I20260808 15:48:41.920333 310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:41.935621 310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:41.940279 310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:41.940620 310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:41.969831 310 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:34021
I20260808 15:48:41.969856 417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:34021 every 8 connection(s)
I20260808 15:48:41.971493 310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:41.972908 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 310
I20260808 15:48:41.973127 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:41.973969 27987 external_mini_cluster.cc:1428] Setting key 89c5a94505291f1f76018ba40e44a3c8
I20260808 15:48:41.976414 418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:41.991631 418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17: Bootstrap starting.
Aug 08 15:48:41 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:41.996053 418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:41.997550 418 log.cc:824] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:42.000808 418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17: No bootstrap required, opened a new log
I20260808 15:48:42.002283 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:41.976457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.27.84.254:34021)
0808 15:48:41.977420 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:41.977437 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:41.979285 (+ 1848us) server_negotiation.cc:407] Connection header received
0808 15:48:41.981467 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:41.981499 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:41.982106 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:41.982649 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:41.983991 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:41.986022 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:41.987484 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:41.988670 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:41.988878 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:41.993294 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:41.993312 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:41.993337 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:41.993404 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:41.997667 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:41.998693 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:41.998700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:41.998707 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:41.998899 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:41.999330 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:41.999342 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:41.999345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:41.999907 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:42.000277 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:42.000713 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 15:48:42.001216 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":394,"threads_started":1}
I20260808 15:48:42.009646 418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8618db3375ee427e8c0874f634316a17" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34021 } }
I20260808 15:48:42.010097 418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:42.010218 418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8618db3375ee427e8c0874f634316a17, State: Initialized, Role: FOLLOWER
I20260808 15:48:42.010946 418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8618db3375ee427e8c0874f634316a17" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34021 } }
I20260808 15:48:42.011235 418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:42.011386 418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:42.011540 418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:42.013751 418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8618db3375ee427e8c0874f634316a17" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34021 } }
I20260808 15:48:42.014219 418 leader_election.cc:302] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8618db3375ee427e8c0874f634316a17; no voters:
I20260808 15:48:42.014959 418 leader_election.cc:288] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:42.015139 429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:42.015974 429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [term 1 LEADER]: Becoming Leader. State: Replica: 8618db3375ee427e8c0874f634316a17, State: Running, Role: LEADER
I20260808 15:48:42.016604 429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8618db3375ee427e8c0874f634316a17" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34021 } }
I20260808 15:48:42.017418 418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:42.019567 431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8618db3375ee427e8c0874f634316a17. Latest consensus state: current_term: 1 leader_uuid: "8618db3375ee427e8c0874f634316a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8618db3375ee427e8c0874f634316a17" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34021 } } }
I20260808 15:48:42.019968 431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:42.019860 430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8618db3375ee427e8c0874f634316a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8618db3375ee427e8c0874f634316a17" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34021 } } }
I20260808 15:48:42.020288 430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:42.022145 437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:42.028002 437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:42.037456 437 catalog_manager.cc:1347] Generated new cluster ID: ca5a44a793a940aa9b9225c31334cf7a
I20260808 15:48:42.037596 437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:42.067926 437 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:42.069355 437 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:42.077469 437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8618db3375ee427e8c0874f634316a17: Generated new TSK 0
I20260808 15:48:42.078318 437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:42.144483 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34021
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:42.495186 455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:42.495527 455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:42.495695 455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:42.503131 455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:42.503294 455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:42.503475 455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:42.513804 455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:42.515301 455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:42.516716 455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:42.531311 462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:42.531391 463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:42.531381 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:42.532207 455 server_base.cc:1034] running on GCE node
I20260808 15:48:42.532879 455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:42.533888 455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:42.535133 455 hybrid_clock.cc:648] HybridClock initialized: now 1786204122535091 us; error 52 us; skew 500 ppm
I20260808 15:48:42.535499 455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:42 dist-test-slave-twwt krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204122, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:42.538972 455 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:42.540467 455 webserver.cc:460] Webserver started at http://127.27.84.193:32803/ using document root <none> and password file <none>
I20260808 15:48:42.541229 455 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:42.541392 455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:42.541718 455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:42.544838 455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "56495854a536421cbf5ee5f8fa6c9a2e"
format_stamp: "Formatted at 2026-08-08 15:48:42 on dist-test-slave-twwt"
server_key: "d909a16e9abde2f8d37418a3a66a99e9"
server_key_iv: "a0b30a3cfc4bdfc2fcbc39bfa070dd40"
server_key_version: "encryptionkey@0"
I20260808 15:48:42.545647 455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "56495854a536421cbf5ee5f8fa6c9a2e"
format_stamp: "Formatted at 2026-08-08 15:48:42 on dist-test-slave-twwt"
server_key: "d909a16e9abde2f8d37418a3a66a99e9"
server_key_iv: "a0b30a3cfc4bdfc2fcbc39bfa070dd40"
server_key_version: "encryptionkey@0"
I20260808 15:48:42.550973 455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:48:42.554744 478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:42.556145 455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:42.556376 455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "56495854a536421cbf5ee5f8fa6c9a2e"
format_stamp: "Formatted at 2026-08-08 15:48:42 on dist-test-slave-twwt"
server_key: "d909a16e9abde2f8d37418a3a66a99e9"
server_key_iv: "a0b30a3cfc4bdfc2fcbc39bfa070dd40"
server_key_version: "encryptionkey@0"
I20260808 15:48:42.556607 455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:42.585502 455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:42.589819 455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:42.590154 455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:42.591262 455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:42.592715 455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:42.592824 455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:42.592989 455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:42.593065 455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:42.628654 455 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43671
I20260808 15:48:42.628695 667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43671 every 8 connection(s)
I20260808 15:48:42.630221 455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:42.640723 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 455
I20260808 15:48:42.640978 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:42.641664 27987 external_mini_cluster.cc:1428] Setting key f3238b44b097c8d2f95e32898c40b3c3
Aug 08 15:48:42 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204122, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:42.651813 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:42.633189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:42853 (local address 127.27.84.254:34021)
0808 15:48:42.633700 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:42.633707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:42.634798 (+ 1091us) server_negotiation.cc:407] Connection header received
0808 15:48:42.636674 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:42.636680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:42.636775 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:42.636903 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:42.638507 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:42.639300 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:42.640949 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:42.641440 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:42.641585 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:42.645727 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:42.645732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:42.645736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:42.645769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:42.648475 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:42.649257 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:42.649263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:42.649266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:42.649407 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:42.649863 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:42.649869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:42.649873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:42.650152 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:42.650319 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:42.651155 (+ 836us) server_negotiation.cc:299] Negotiation successful
0808 15:48:42.651413 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":238,"threads_started":1}
I20260808 15:48:42.653249 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:42.633817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34021 (local address 127.27.84.193:42853)
0808 15:48:42.634567 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:42.634678 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:42.636366 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:42.637174 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:42.637183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:42.637713 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:42.638319 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:42.638336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:42.639476 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:42.639483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:42.640770 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:42.640787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:42.641690 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:42.641697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:42.641954 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:42.642738 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:42.642762 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:42.645492 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:42.648635 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:42.648645 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:42.648666 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:42.649126 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:42.649527 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:42.649535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:42.649539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:42.649742 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:42.650292 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:42.650300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:42.650747 (+ 447us) client_negotiation.cc:769] Sending connection context
0808 15:48:42.651038 (+ 291us) client_negotiation.cc:240] Negotiation successful
0808 15:48:42.651630 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":288,"threads_started":1}
I20260808 15:48:42.655151 671 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34021
I20260808 15:48:42.655489 671 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:42.656275 671 heartbeater.cc:511] Master 127.27.84.254:34021 requested a full tablet report, sending...
I20260808 15:48:42.658860 366 ts_manager.cc:194] Registered new tserver with Master: 56495854a536421cbf5ee5f8fa6c9a2e (127.27.84.193:43671)
I20260808 15:48:42.661170 366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:42853
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:42.699120 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34021
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:43.070343 679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:43.070730 679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:43.070848 679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:43.079167 679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:43.079291 679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:43.079416 679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:43.089543 679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:43.091253 679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:43.092612 679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:43.106509 690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:43.106567 692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:43.106617 694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:43.107422 679 server_base.cc:1034] running on GCE node
I20260808 15:48:43.108063 679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:43.109063 679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:43.110306 679 hybrid_clock.cc:648] HybridClock initialized: now 1786204123110256 us; error 72 us; skew 500 ppm
I20260808 15:48:43.110705 679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:43.113994 679 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:43.115585 679 webserver.cc:460] Webserver started at http://127.27.84.194:38417/ using document root <none> and password file <none>
I20260808 15:48:43.116250 679 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:43.116356 679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:43.116608 679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:43.119869 679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "e4dd2dedf4c340f2ac031cb1b7f2b208"
format_stamp: "Formatted at 2026-08-08 15:48:43 on dist-test-slave-twwt"
server_key: "d7c5dfba1e8d5c1f325d86aac5b0ada9"
server_key_iv: "404b967365cffcb8b90671a096461052"
server_key_version: "encryptionkey@0"
I20260808 15:48:43.120636 679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "e4dd2dedf4c340f2ac031cb1b7f2b208"
format_stamp: "Formatted at 2026-08-08 15:48:43 on dist-test-slave-twwt"
server_key: "d7c5dfba1e8d5c1f325d86aac5b0ada9"
server_key_iv: "404b967365cffcb8b90671a096461052"
server_key_version: "encryptionkey@0"
I20260808 15:48:43.126576 679 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260808 15:48:43.130304 701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:43.131878 679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:43.132076 679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "e4dd2dedf4c340f2ac031cb1b7f2b208"
format_stamp: "Formatted at 2026-08-08 15:48:43 on dist-test-slave-twwt"
server_key: "d7c5dfba1e8d5c1f325d86aac5b0ada9"
server_key_iv: "404b967365cffcb8b90671a096461052"
server_key_version: "encryptionkey@0"
I20260808 15:48:43.132278 679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:43.146193 679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:43.150424 679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:43.150801 679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:43.151893 679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:43.153519 679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:43.153597 679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:43.153781 679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:43.153872 679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:43.189586 679 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:35273
I20260808 15:48:43.189635 878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:35273 every 8 connection(s)
I20260808 15:48:43.191038 679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:43.196659 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 679
I20260808 15:48:43.196828 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:43.197325 27987 external_mini_cluster.cc:1428] Setting key fdeff59034a776351877ac80ef9a8783
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:43.212158 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.193809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:52061 (local address 127.27.84.254:34021)
0808 15:48:43.194265 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:43.194270 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:43.195449 (+ 1179us) server_negotiation.cc:407] Connection header received
0808 15:48:43.197464 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:43.197470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:43.197563 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:43.197700 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:43.199329 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.200059 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.201650 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.202088 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.202230 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.205956 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:43.205967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:43.205970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:43.206005 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:43.208804 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.209592 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.209597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.209599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.209736 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.210233 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.210243 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.210245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.210528 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:43.210727 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:43.211504 (+ 777us) server_negotiation.cc:299] Negotiation successful
0808 15:48:43.211757 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
I20260808 15:48:43.213482 883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.194436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34021 (local address 127.27.84.194:52061)
0808 15:48:43.195284 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:43.195328 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:43.197180 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:43.197958 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:43.197967 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:43.198482 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:43.199141 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:43.199160 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.200231 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.200235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:43.201491 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:43.201504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.202343 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.202351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:43.202600 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.203414 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:43.203438 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:43.205693 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:43.208976 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:43.208985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:43.209004 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:43.209446 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:43.209867 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:43.209871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:43.209875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:43.210099 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:43.210716 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:43.210725 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:43.211107 (+ 382us) client_negotiation.cc:769] Sending connection context
0808 15:48:43.211387 (+ 280us) client_negotiation.cc:240] Negotiation successful
0808 15:48:43.212014 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":405,"threads_started":1}
I20260808 15:48:43.215232 879 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34021
I20260808 15:48:43.215627 879 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:43.216691 879 heartbeater.cc:511] Master 127.27.84.254:34021 requested a full tablet report, sending...
I20260808 15:48:43.218735 366 ts_manager.cc:194] Registered new tserver with Master: e4dd2dedf4c340f2ac031cb1b7f2b208 (127.27.84.194:35273)
I20260808 15:48:43.220081 366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:52061
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:43.254994 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34021
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:43.602944 886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:43.603283 886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:43.603464 886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:43.611100 886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:43.611289 886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:43.611485 886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:43.621937 886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:43.623404 886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:43.624908 886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:43.639257 895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:43.639287 894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:43.639205 897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:43.640312 886 server_base.cc:1034] running on GCE node
I20260808 15:48:43.641013 886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:43.642087 886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:43.643349 886 hybrid_clock.cc:648] HybridClock initialized: now 1786204123643288 us; error 69 us; skew 500 ppm
I20260808 15:48:43.643707 886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:43.647231 886 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:43.648752 886 webserver.cc:460] Webserver started at http://127.27.84.195:40875/ using document root <none> and password file <none>
I20260808 15:48:43.649488 886 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:43.649644 886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:43.649982 886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:43.653127 886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "0707935585d74e8ab89338d390e421f6"
format_stamp: "Formatted at 2026-08-08 15:48:43 on dist-test-slave-twwt"
server_key: "4205598fcbbc77b04a51288fca2eb022"
server_key_iv: "db4b1176a6dd9597ee492a8190e5040b"
server_key_version: "encryptionkey@0"
I20260808 15:48:43.653970 886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "0707935585d74e8ab89338d390e421f6"
format_stamp: "Formatted at 2026-08-08 15:48:43 on dist-test-slave-twwt"
server_key: "4205598fcbbc77b04a51288fca2eb022"
server_key_iv: "db4b1176a6dd9597ee492a8190e5040b"
server_key_version: "encryptionkey@0"
I20260808 15:48:43.659420 886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:48:43.663350 910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:43.664510 671 heartbeater.cc:503] Master 127.27.84.254:34021 was elected leader, sending a full tablet report...
I20260808 15:48:43.665064 886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:43.665263 886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "0707935585d74e8ab89338d390e421f6"
format_stamp: "Formatted at 2026-08-08 15:48:43 on dist-test-slave-twwt"
server_key: "4205598fcbbc77b04a51288fca2eb022"
server_key_iv: "db4b1176a6dd9597ee492a8190e5040b"
server_key_version: "encryptionkey@0"
I20260808 15:48:43.665515 886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:43.689721 886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:43.694391 886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:43.694732 886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:43.695695 886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:43.697037 886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:43.697113 886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:43.697197 886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:43.697249 886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:43.733945 886 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:46597
I20260808 15:48:43.734040 1084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:46597 every 8 connection(s)
I20260808 15:48:43.735420 886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:43.739789 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 886
I20260808 15:48:43.740007 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:43.740593 27987 external_mini_cluster.cc:1428] Setting key 682f73a5e1965d9a607b02a5e0049a08
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:43.757223 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.738298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:55403 (local address 127.27.84.254:34021)
0808 15:48:43.738829 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:43.738835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:43.740116 (+ 1281us) server_negotiation.cc:407] Connection header received
0808 15:48:43.742343 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:43.742348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:43.742416 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:43.742550 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:43.744344 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.745175 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.746769 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.747188 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.747286 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.750799 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:43.750803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:43.750807 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:43.750865 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:43.753637 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.754377 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.754382 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.754384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.754551 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.755155 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.755159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.755161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.755455 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:43.755578 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:43.756624 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0808 15:48:43.756934 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":232,"threads_started":1}
I20260808 15:48:43.758103 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.738976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34021 (local address 127.27.84.195:55403)
0808 15:48:43.739863 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:43.739934 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:43.742025 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:43.742867 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:43.742881 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:43.743422 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:43.744180 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:43.744199 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.745310 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.745314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:43.746587 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:43.746600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.747390 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.747395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:43.747593 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.748235 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:43.748258 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:43.750532 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:43.753760 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:43.753767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:43.753779 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:43.754230 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:43.754726 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:43.754732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:43.754735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:43.755019 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:43.755597 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:43.755606 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:43.756068 (+ 462us) client_negotiation.cc:769] Sending connection context
0808 15:48:43.756422 (+ 354us) client_negotiation.cc:240] Negotiation successful
0808 15:48:43.756868 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":331,"threads_started":1}
I20260808 15:48:43.759662 1086 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34021
I20260808 15:48:43.760013 1086 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:43.760885 1086 heartbeater.cc:511] Master 127.27.84.254:34021 requested a full tablet report, sending...
I20260808 15:48:43.762523 366 ts_manager.cc:194] Registered new tserver with Master: 0707935585d74e8ab89338d390e421f6 (127.27.84.195:46597)
I20260808 15:48:43.763489 366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:55403
I20260808 15:48:43.768822 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:43.829502 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.814395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56492 (local address 127.27.84.254:34021)
0808 15:48:43.814708 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:43.814718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:43.815099 (+ 381us) server_negotiation.cc:407] Connection header received
0808 15:48:43.815343 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:43.815347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:43.815420 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:43.815536 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:43.816970 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.817780 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.819184 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.819604 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.819753 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.823984 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:43.823990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:43.823993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:43.824032 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:43.826767 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.827439 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.827444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.827448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.827582 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.827991 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.827998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.828006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.828340 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:43.828604 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:43.828927 (+ 323us) server_negotiation.cc:299] Negotiation successful
0808 15:48:43.829171 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 15:48:43.830088 366 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56492
I20260808 15:48:43.832257 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 455
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:43.848855 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.833482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.27.84.193:43671)
0808 15:48:43.834183 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:43.834191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:43.834216 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:48:43.834273 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:43.834280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:43.834511 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:43.834705 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:43.835925 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.836996 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.838184 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.838851 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.838999 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.843105 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:43.843116 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:43.843132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:43.843171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:43.845947 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.846682 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.846690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.846693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.846826 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.847208 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.847213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.847217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.847600 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:43.847919 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:43.848217 (+ 298us) server_negotiation.cc:299] Negotiation successful
0808 15:48:43.848501 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":367,"threads_started":1}
W20260808 15:48:43.850113 661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35092
W20260808 15:48:43.851289 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 455: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:43.851526 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 455
I20260808 15:48:43.875227 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 679
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:43.891999 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.876243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.27.84.194:35273)
0808 15:48:43.876895 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:43.876902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:43.876931 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:48:43.876999 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:43.877006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:43.877239 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:43.877383 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:43.878582 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.879730 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.880966 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.881524 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.881625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.885996 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:43.886009 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:43.886025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:43.886063 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:43.889156 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.889878 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.889886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.889888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.890022 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.890365 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.890368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.890370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.890799 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:43.891084 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:43.891346 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 15:48:43.891599 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":353,"threads_started":1}
W20260808 15:48:43.893395 872 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39680
W20260808 15:48:43.894840 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 679: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:43.895016 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 679
I20260808 15:48:43.927484 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 886
Aug 08 15:48:43 dist-test-slave-twwt krb5kdc[32756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204123, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:43.945483 1108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:43.928606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.27.84.195:46597)
0808 15:48:43.929379 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:43.929386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:43.929415 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:48:43.929480 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:43.929487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:43.929708 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:43.929844 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:43.931256 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.932369 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.933594 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:43.934181 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:43.934284 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:43.939079 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:43.939117 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:43.939140 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:43.939195 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:43.942170 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.943266 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.943274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.943277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.943423 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:43.943885 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:43.943890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:43.943892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:43.944312 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:43.944571 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:43.944941 (+ 370us) server_negotiation.cc:299] Negotiation successful
0808 15:48:43.945168 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":422,"threads_started":1}
W20260808 15:48:43.947098 1079 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53604
W20260808 15:48:43.948355 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 886: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:48:43.948530 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 886
I20260808 15:48:43.984043 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 310
I20260808 15:48:43.985507 412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:44.092468 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 310
2026-08-08T15:48:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4154 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:44 dist-test-slave-twwt krb5kdc[1114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:44 dist-test-slave-twwt krb5kdc[1114](info): set up 2 sockets
Aug 08 15:48:44 dist-test-slave-twwt krb5kdc[1114](info): commencing operation
krb5kdc: starting...
W20260808 15:48:45.484107 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:45 dist-test-slave-twwt krb5kdc[1114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:45.743342 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44107
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:46.100169 1132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:46.100493 1132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:46.100662 1132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:46.108033 1132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:46.108139 1132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:46.108268 1132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:46.108376 1132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:46.108462 1132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:46.118448 1132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:46.119841 1132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:46.121228 1132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:46.133575 1141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:46.133548 1142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:46.133635 1144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:46.134284 1132 server_base.cc:1034] running on GCE node
I20260808 15:48:46.135183 1132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:46.136686 1132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:46.137923 1132 hybrid_clock.cc:648] HybridClock initialized: now 1786204126137879 us; error 66 us; skew 500 ppm
I20260808 15:48:46.138283 1132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:46 dist-test-slave-twwt krb5kdc[1114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:46.146868 1132 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:46.148492 1132 webserver.cc:460] Webserver started at http://127.27.84.254:40925/ using document root <none> and password file <none>
I20260808 15:48:46.149166 1132 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:46.149271 1132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:46.149576 1132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:46.152779 1132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "ea4a2e875cb34e8fb083d65520864e37"
format_stamp: "Formatted at 2026-08-08 15:48:46 on dist-test-slave-twwt"
server_key: "3d3d1dea9d9cabac396640dab632df2a"
server_key_iv: "465e69170ed66833ab3d0c4b132b5e2f"
server_key_version: "encryptionkey@0"
I20260808 15:48:46.153607 1132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "ea4a2e875cb34e8fb083d65520864e37"
format_stamp: "Formatted at 2026-08-08 15:48:46 on dist-test-slave-twwt"
server_key: "3d3d1dea9d9cabac396640dab632df2a"
server_key_iv: "465e69170ed66833ab3d0c4b132b5e2f"
server_key_version: "encryptionkey@0"
I20260808 15:48:46.159068 1132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:48:46.162757 1156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:46.164315 1132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:46.164525 1132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "ea4a2e875cb34e8fb083d65520864e37"
format_stamp: "Formatted at 2026-08-08 15:48:46 on dist-test-slave-twwt"
server_key: "3d3d1dea9d9cabac396640dab632df2a"
server_key_iv: "465e69170ed66833ab3d0c4b132b5e2f"
server_key_version: "encryptionkey@0"
I20260808 15:48:46.164749 1132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:46.200601 1132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:46.204890 1132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:46.205185 1132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:46.225337 1132 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:44107
I20260808 15:48:46.225378 1240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:44107 every 8 connection(s)
I20260808 15:48:46.226933 1132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:46.229723 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 1132
I20260808 15:48:46.229982 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:46.230751 27987 external_mini_cluster.cc:1428] Setting key 171737c0b7b68186134c6af09c18f500
I20260808 15:48:46.231071 1241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:46.244257 1241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37: Bootstrap starting.
Aug 08 15:48:46 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:46.247783 1241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:46.249024 1241 log.cc:824] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:46.252009 1241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37: No bootstrap required, opened a new log
I20260808 15:48:46.254379 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:46.233289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.27.84.254:44107)
0808 15:48:46.234100 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:46.234111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:46.234155 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:48:46.236268 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:46.236291 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:46.236700 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:46.237148 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:46.238698 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:46.239971 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:46.241208 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:46.241927 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:46.242143 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:46.246348 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:46.246356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:46.246371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:46.246415 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:46.249827 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:46.250529 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:46.250536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:46.250543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:46.250773 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:46.251163 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:46.251168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:46.251172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:46.251691 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:46.252004 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:46.252510 (+ 506us) server_negotiation.cc:299] Negotiation successful
0808 15:48:46.253010 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":354,"threads_started":1}
I20260808 15:48:46.261179 1241 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4a2e875cb34e8fb083d65520864e37" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44107 } }
I20260808 15:48:46.261545 1241 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:46.261626 1241 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4a2e875cb34e8fb083d65520864e37, State: Initialized, Role: FOLLOWER
I20260808 15:48:46.262204 1241 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4a2e875cb34e8fb083d65520864e37" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44107 } }
I20260808 15:48:46.262418 1241 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:46.262517 1241 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:46.262712 1241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:46.265156 1241 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4a2e875cb34e8fb083d65520864e37" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44107 } }
I20260808 15:48:46.265632 1241 leader_election.cc:302] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4a2e875cb34e8fb083d65520864e37; no voters:
I20260808 15:48:46.266201 1241 leader_election.cc:288] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:46.266366 1247 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:46.267169 1247 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [term 1 LEADER]: Becoming Leader. State: Replica: ea4a2e875cb34e8fb083d65520864e37, State: Running, Role: LEADER
I20260808 15:48:46.267737 1247 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4a2e875cb34e8fb083d65520864e37" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44107 } }
I20260808 15:48:46.268546 1241 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:46.275434 1249 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4a2e875cb34e8fb083d65520864e37. Latest consensus state: current_term: 1 leader_uuid: "ea4a2e875cb34e8fb083d65520864e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4a2e875cb34e8fb083d65520864e37" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44107 } } }
I20260808 15:48:46.275977 1249 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:46.278959 1248 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea4a2e875cb34e8fb083d65520864e37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4a2e875cb34e8fb083d65520864e37" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44107 } } }
I20260808 15:48:46.279543 1248 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37 [sys.catalog]: This master's current role is: LEADER
W20260808 15:48:46.283921 1270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:48:46.284062 1270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:48:46.284202 1272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:46.286011 1272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:46.294721 1272 catalog_manager.cc:1347] Generated new cluster ID: c9d64b32b0e741688024cda75ed9e79a
I20260808 15:48:46.294850 1272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:46.311478 1272 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:46.312831 1272 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:46.323046 1272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4a2e875cb34e8fb083d65520864e37: Generated new TSK 0
I20260808 15:48:46.323976 1272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:46.383299 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44107
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:46.739485 1278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:46.739794 1278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:46.739892 1278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:46.747581 1278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:46.747704 1278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:46.747829 1278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:46.758180 1278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:46.759616 1278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:46.760996 1278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:46.775341 1291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:46.775341 1286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:46.775796 1284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:46.776350 1278 server_base.cc:1034] running on GCE node
I20260808 15:48:46.777124 1278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:46.778219 1278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:46.779466 1278 hybrid_clock.cc:648] HybridClock initialized: now 1786204126779404 us; error 55 us; skew 500 ppm
I20260808 15:48:46.779917 1278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:46 dist-test-slave-twwt krb5kdc[1114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:46.783310 1278 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:46.784859 1278 webserver.cc:460] Webserver started at http://127.27.84.193:38823/ using document root <none> and password file <none>
I20260808 15:48:46.785562 1278 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:46.785665 1278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:46.785940 1278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:46.788987 1278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "645b5cbd27274e81b6612fa04fa20a35"
format_stamp: "Formatted at 2026-08-08 15:48:46 on dist-test-slave-twwt"
server_key: "2b4846d33e5bb9e2fccd5296879a9409"
server_key_iv: "fd7d01f12d46b3cd5f5d98e0ed0484c5"
server_key_version: "encryptionkey@0"
I20260808 15:48:46.789754 1278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "645b5cbd27274e81b6612fa04fa20a35"
format_stamp: "Formatted at 2026-08-08 15:48:46 on dist-test-slave-twwt"
server_key: "2b4846d33e5bb9e2fccd5296879a9409"
server_key_iv: "fd7d01f12d46b3cd5f5d98e0ed0484c5"
server_key_version: "encryptionkey@0"
I20260808 15:48:46.795233 1278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:48:46.798753 1302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:46.800112 1278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:46.800289 1278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "645b5cbd27274e81b6612fa04fa20a35"
format_stamp: "Formatted at 2026-08-08 15:48:46 on dist-test-slave-twwt"
server_key: "2b4846d33e5bb9e2fccd5296879a9409"
server_key_iv: "fd7d01f12d46b3cd5f5d98e0ed0484c5"
server_key_version: "encryptionkey@0"
I20260808 15:48:46.800489 1278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:46.833122 1278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:46.837388 1278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:46.837707 1278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:46.838810 1278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:46.840243 1278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:46.840319 1278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:46.840514 1278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:46.840569 1278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:46.875279 1278 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:37983
I20260808 15:48:46.875372 1477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:37983 every 8 connection(s)
I20260808 15:48:46.876667 1278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:46.880081 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 1278
I20260808 15:48:46.880342 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:46.881208 27987 external_mini_cluster.cc:1428] Setting key 01626cf9147193c8d6e778bcadb0be23
Aug 08 15:48:46 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:46.899976 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:46.879351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:54725 (local address 127.27.84.254:44107)
0808 15:48:46.879851 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:46.879855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:46.880975 (+ 1120us) server_negotiation.cc:407] Connection header received
0808 15:48:46.883101 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:46.883113 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:46.883193 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:46.883316 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:46.885419 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:46.886121 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:46.888642 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:46.889042 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:46.889137 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:46.893807 (+ 4670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:46.893815 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:46.893824 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:46.893864 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:46.896520 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:46.897252 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:46.897258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:46.897261 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:46.897394 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:46.897827 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:46.897832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:46.897835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:46.898251 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:46.898400 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:46.899372 (+ 972us) server_negotiation.cc:299] Negotiation successful
0808 15:48:46.899600 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":277,"threads_started":1}
I20260808 15:48:46.901005 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:46.879971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44107 (local address 127.27.84.193:54725)
0808 15:48:46.880805 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:46.880844 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:46.882787 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:46.883613 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:46.883625 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:46.884417 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:46.885243 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:46.885272 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:46.886308 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:46.886321 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:46.888490 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:46.888509 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:46.889276 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:46.889285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:46.889630 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:46.890386 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:46.890428 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:46.893567 (+ 3139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:46.896668 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:46.896677 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:46.896693 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:46.897116 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:46.897508 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:46.897512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:46.897516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:46.897704 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:46.898417 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:46.898431 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:46.898988 (+ 557us) client_negotiation.cc:769] Sending connection context
0808 15:48:46.899261 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 15:48:46.899801 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":381,"threads_started":1}
I20260808 15:48:46.902887 1479 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44107
I20260808 15:48:46.903228 1479 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:46.904057 1479 heartbeater.cc:511] Master 127.27.84.254:44107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
I20260808 15:48:46.906733 1181 ts_manager.cc:194] Registered new tserver with Master: 645b5cbd27274e81b6612fa04fa20a35 (127.27.84.193:37983)
I20260808 15:48:46.908361 1181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:54725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:46.937968 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44107
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:47.282595 1487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:47.282953 1487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:47.283052 1487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:47.290524 1487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:47.290740 1487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:47.290879 1487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:47.301286 1487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:47.302767 1487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:47.304291 1487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:47.318028 1494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:47.318086 1497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:47.318109 1493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:47.318920 1487 server_base.cc:1034] running on GCE node
I20260808 15:48:47.319525 1487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:47.320446 1487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:47.321687 1487 hybrid_clock.cc:648] HybridClock initialized: now 1786204127321589 us; error 79 us; skew 500 ppm
I20260808 15:48:47.322101 1487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:47 dist-test-slave-twwt krb5kdc[1114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:47.325721 1487 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:47.327401 1487 webserver.cc:460] Webserver started at http://127.27.84.194:40495/ using document root <none> and password file <none>
I20260808 15:48:47.328133 1487 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:47.328244 1487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:47.328521 1487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:47.331836 1487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "991bb415851d4608990123216005643e"
format_stamp: "Formatted at 2026-08-08 15:48:47 on dist-test-slave-twwt"
server_key: "d72757b7a1e9925883db71b551632e83"
server_key_iv: "ef578ea4d8c258390a28cdd6ef2153a9"
server_key_version: "encryptionkey@0"
I20260808 15:48:47.332782 1487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "991bb415851d4608990123216005643e"
format_stamp: "Formatted at 2026-08-08 15:48:47 on dist-test-slave-twwt"
server_key: "d72757b7a1e9925883db71b551632e83"
server_key_iv: "ef578ea4d8c258390a28cdd6ef2153a9"
server_key_version: "encryptionkey@0"
I20260808 15:48:47.338569 1487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:48:47.342468 1507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:47.343966 1487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:47.344167 1487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "991bb415851d4608990123216005643e"
format_stamp: "Formatted at 2026-08-08 15:48:47 on dist-test-slave-twwt"
server_key: "d72757b7a1e9925883db71b551632e83"
server_key_iv: "ef578ea4d8c258390a28cdd6ef2153a9"
server_key_version: "encryptionkey@0"
I20260808 15:48:47.344399 1487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:47.365293 1487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:47.370152 1487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:47.370455 1487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:47.371575 1487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:47.373111 1487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:47.373226 1487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:47.373385 1487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:47.373461 1487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:47.411497 1487 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:38613
I20260808 15:48:47.411559 1679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:38613 every 8 connection(s)
I20260808 15:48:47.412921 1487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:47.414151 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 1487
I20260808 15:48:47.414361 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:47.415024 27987 external_mini_cluster.cc:1428] Setting key fd0d7d9d8bc3b872a9f15b9f7b4904a9
Aug 08 15:48:47 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:47.435199 1685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:47.416000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:40353 (local address 127.27.84.254:44107)
0808 15:48:47.417859 (+ 1859us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:47.417865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:47.417886 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:48:47.419541 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:47.419550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:47.419644 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:47.419767 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:47.421350 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.422504 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.424178 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.424923 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.425066 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:47.429169 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:47.429174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:47.429177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:47.429214 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:47.432025 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:47.432799 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:47.432809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:47.432812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:47.432944 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:47.433403 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:47.433407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:47.433409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:47.433683 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:47.433844 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:47.434561 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 15:48:47.434818 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1749,"thread_start_us":2647,"threads_started":1}
I20260808 15:48:47.436326 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:47.416552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44107 (local address 127.27.84.194:40353)
0808 15:48:47.417396 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:47.417450 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:47.419265 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:47.420010 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:47.420019 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:47.420552 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:47.421164 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:47.421187 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.422714 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.422718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:47.424023 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:47.424037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.425153 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.425158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:47.425370 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:47.426082 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:47.426105 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:47.428923 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:47.432175 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:47.432185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:47.432202 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:47.432648 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:47.433062 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:47.433066 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:47.433070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:47.433266 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:47.433819 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:47.433826 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:47.434190 (+ 364us) client_negotiation.cc:769] Sending connection context
0808 15:48:47.434463 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 15:48:47.435053 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":350,"threads_started":1}
I20260808 15:48:47.438042 1680 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44107
I20260808 15:48:47.438464 1680 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:47.439517 1680 heartbeater.cc:511] Master 127.27.84.254:44107 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
I20260808 15:48:47.441378 1181 ts_manager.cc:194] Registered new tserver with Master: 991bb415851d4608990123216005643e (127.27.84.194:38613)
I20260808 15:48:47.442348 1181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:40353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:47.474066 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44107
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:47.828133 1692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:47.828452 1692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:47.828617 1692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:47.836061 1692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:47.836186 1692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:47.836373 1692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:47.846639 1692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:47.848206 1692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:47.849716 1692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:47.863796 1700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:47.863830 1701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:47.863768 1703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:47.864686 1692 server_base.cc:1034] running on GCE node
I20260808 15:48:47.865356 1692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:47.866416 1692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:47.867666 1692 hybrid_clock.cc:648] HybridClock initialized: now 1786204127867607 us; error 46 us; skew 500 ppm
I20260808 15:48:47.868050 1692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:47 dist-test-slave-twwt krb5kdc[1114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:47.871860 1692 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:47.873631 1692 webserver.cc:460] Webserver started at http://127.27.84.195:35771/ using document root <none> and password file <none>
I20260808 15:48:47.874380 1692 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:47.874527 1692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:47.874948 1692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:47.877988 1692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "a5a4951fc3354fc7af54e6a378fc165b"
format_stamp: "Formatted at 2026-08-08 15:48:47 on dist-test-slave-twwt"
server_key: "9aef740f2e04556f9f9881a87e7ca9ea"
server_key_iv: "e0b36e02d5334ff2a6474d66236f4515"
server_key_version: "encryptionkey@0"
I20260808 15:48:47.878973 1692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "a5a4951fc3354fc7af54e6a378fc165b"
format_stamp: "Formatted at 2026-08-08 15:48:47 on dist-test-slave-twwt"
server_key: "9aef740f2e04556f9f9881a87e7ca9ea"
server_key_iv: "e0b36e02d5334ff2a6474d66236f4515"
server_key_version: "encryptionkey@0"
I20260808 15:48:47.884387 1692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:48:47.888074 1714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:47.889452 1692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:47.889634 1692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "a5a4951fc3354fc7af54e6a378fc165b"
format_stamp: "Formatted at 2026-08-08 15:48:47 on dist-test-slave-twwt"
server_key: "9aef740f2e04556f9f9881a87e7ca9ea"
server_key_iv: "e0b36e02d5334ff2a6474d66236f4515"
server_key_version: "encryptionkey@0"
I20260808 15:48:47.889854 1692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:47.901998 1692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:47.906783 1692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:47.907109 1692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:47.908155 1692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:47.909591 1692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:47.909700 1692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:47.909850 1692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:47.909929 1692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:47.911574 1479 heartbeater.cc:503] Master 127.27.84.254:44107 was elected leader, sending a full tablet report...
I20260808 15:48:47.944914 1692 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:45619
I20260808 15:48:47.944979 1890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:45619 every 8 connection(s)
I20260808 15:48:47.946259 1692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:47.950043 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 1692
I20260808 15:48:47.950297 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:47.951123 27987 external_mini_cluster.cc:1428] Setting key b0c55e25042e7f45b5b2ab82545683c0
Aug 08 15:48:47 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:47.971490 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:47.949238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:57585 (local address 127.27.84.254:44107)
0808 15:48:47.949864 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:47.949871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:47.953466 (+ 3595us) server_negotiation.cc:407] Connection header received
0808 15:48:47.953546 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:47.953554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:47.953647 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:47.953756 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:47.955853 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.957137 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.959453 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.960180 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.960344 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:47.964198 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:47.964205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:47.964208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:47.964244 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:47.967792 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:47.968591 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:47.968596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:47.968598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:47.968789 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:47.969355 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:47.969372 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:47.969374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:47.969718 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:47.969890 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:47.970717 (+ 827us) server_negotiation.cc:299] Negotiation successful
0808 15:48:47.971073 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":220,"threads_started":1}
I20260808 15:48:47.972276 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:47.949861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44107 (local address 127.27.84.195:57585)
0808 15:48:47.950837 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:47.950883 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:47.953089 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:47.954112 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:47.954121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:47.954777 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:47.955632 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:47.955661 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.957386 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.957391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:47.959285 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:47.959310 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:47.960542 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:47.960551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:47.960770 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:47.961489 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:47.961513 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:47.963943 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:47.967934 (+ 3991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:47.967941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:47.967954 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:47.968466 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:47.968930 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:47.968934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:47.968937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:47.969194 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:47.969868 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:47.969875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:47.970221 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 15:48:47.970471 (+ 250us) client_negotiation.cc:240] Negotiation successful
0808 15:48:47.970998 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":421,"threads_started":1}
I20260808 15:48:47.973767 1891 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44107
I20260808 15:48:47.974115 1891 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:47.974934 1891 heartbeater.cc:511] Master 127.27.84.254:44107 requested a full tablet report, sending...
I20260808 15:48:47.976423 1181 ts_manager.cc:194] Registered new tserver with Master: a5a4951fc3354fc7af54e6a378fc165b (127.27.84.195:45619)
I20260808 15:48:47.977202 1181 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:57585
I20260808 15:48:47.980450 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:48.042418 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:48.025923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.27.84.193:37983)
0808 15:48:48.026708 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:48.026717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:48.026745 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:48:48.026809 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:48.026816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:48.027051 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:48.027243 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:48.028910 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.029990 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.031390 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.032039 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.032197 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:48.036470 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:48.036485 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:48.036502 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:48.036545 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:48.039681 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.040440 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.040448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.040451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.040603 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.040968 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.040972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.040974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.041381 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:48.041629 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:48.041876 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 15:48:48.042091 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":363,"threads_started":1}
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:48.062045 1897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:48.047637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.27.84.254:44107)
0808 15:48:48.047888 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:48.047894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:48.048152 (+ 258us) server_negotiation.cc:407] Connection header received
0808 15:48:48.048325 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:48.048331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:48.048453 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:48.048575 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:48.050186 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.051061 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.052320 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.052753 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.052854 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:48.056736 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:48.056742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:48.056745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:48.056780 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:48.059537 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.060193 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.060198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.060200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.060357 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.060697 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.060701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.060703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.061043 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:48.061232 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:48.061511 (+ 279us) server_negotiation.cc:299] Negotiation successful
0808 15:48:48.061687 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260808 15:48:48.062682 1181 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53200
I20260808 15:48:48.063758 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 1278
I20260808 15:48:48.076489 1906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:48.064744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.27.84.193:37983)
0808 15:48:48.064926 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:48.064930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:48.065167 (+ 237us) server_negotiation.cc:407] Connection header received
0808 15:48:48.065282 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:48.065286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:48.065392 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:48.065493 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:48.066864 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.067541 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.068812 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.069249 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.069422 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:48.071750 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:48.071754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:48.071757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:48.071790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:48.074353 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.074993 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.074997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.075000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.075117 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.075444 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.075447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.075449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.075678 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:48.075849 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:48.076101 (+ 252us) server_negotiation.cc:299] Negotiation successful
0808 15:48:48.076257 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:48:48.077134 1470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:48.161556 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 1278
I20260808 15:48:48.189111 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 1487
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:48.206066 1913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:48.190115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.27.84.194:38613)
0808 15:48:48.190811 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:48.190835 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:48.190878 (+ 43us) server_negotiation.cc:407] Connection header received
0808 15:48:48.190958 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:48.190972 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:48.191195 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:48.191325 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:48.192645 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.193693 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.195220 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.195798 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.195920 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:48.200405 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:48.200419 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:48.200434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:48.200477 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:48.203366 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.204093 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.204100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.204102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.204231 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.204606 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.204609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.204611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.204952 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:48.205179 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:48.205448 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 15:48:48.205646 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":316,"threads_started":1}
I20260808 15:48:48.207595 1674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:48.291503 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 1487
I20260808 15:48:48.315903 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 1692
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:48.333914 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:48.317153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.27.84.195:45619)
0808 15:48:48.317821 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:48.317828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:48.317857 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:48:48.317919 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:48.317925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:48.318138 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:48.318263 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:48.319789 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.320849 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.322320 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:48.323073 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:48.323185 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:48.328186 (+ 5001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:48.328201 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:48.328218 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:48.328257 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:48.331213 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.331917 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.331924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.331927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.332051 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:48.332495 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:48.332498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:48.332500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:48.332867 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:48.333105 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:48.333378 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 15:48:48.333618 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":343,"threads_started":1}
I20260808 15:48:48.335327 1885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:48.421650 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 1692
I20260808 15:48:48.448288 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 1132
I20260808 15:48:48.449677 1232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:48.556026 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 1132
2026-08-08T15:48:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4464 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1925](info): set up 2 sockets
Aug 08 15:48:48 dist-test-slave-twwt krb5kdc[1925](info): commencing operation
krb5kdc: starting...
W20260808 15:48:49.881119 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.272s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:48:50 dist-test-slave-twwt krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:48:50.136036 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41825
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:41825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:50.475086 1951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:50.475432 1951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:50.475602 1951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:50.482843 1951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:50.482991 1951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:50.483093 1951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:50.483192 1951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:50.483278 1951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:50.493289 1951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:41825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:50.494699 1951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:50.496110 1951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:50.507984 1957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:50.507987 1964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:50.508010 1958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:50.508965 1951 server_base.cc:1034] running on GCE node
I20260808 15:48:50.509707 1951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:50.511176 1951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:50.512373 1951 hybrid_clock.cc:648] HybridClock initialized: now 1786204130512314 us; error 51 us; skew 500 ppm
I20260808 15:48:50.512758 1951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:50 dist-test-slave-twwt krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204130, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:50.516001 1951 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:48:50.517282 1951 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:50.517377 1951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:50.517666 1951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:50.520874 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a"
format_stamp: "Formatted at 2026-08-08 15:48:50 on dist-test-slave-twwt"
server_key: "a13ea8728c57a4df66d113b5af81f37a"
server_key_iv: "462876d1bf9318f081e9b4125e0a890f"
server_key_version: "encryptionkey@0"
I20260808 15:48:50.521670 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a"
format_stamp: "Formatted at 2026-08-08 15:48:50 on dist-test-slave-twwt"
server_key: "a13ea8728c57a4df66d113b5af81f37a"
server_key_iv: "462876d1bf9318f081e9b4125e0a890f"
server_key_version: "encryptionkey@0"
I20260808 15:48:50.527010 1951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:48:50.530565 1976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:50.532043 1951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:50.532253 1951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a"
format_stamp: "Formatted at 2026-08-08 15:48:50 on dist-test-slave-twwt"
server_key: "a13ea8728c57a4df66d113b5af81f37a"
server_key_iv: "462876d1bf9318f081e9b4125e0a890f"
server_key_version: "encryptionkey@0"
I20260808 15:48:50.532507 1951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:50.546590 1951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:50.550705 1951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:50.550971 1951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:50.570099 1951 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:41825
I20260808 15:48:50.570118 2051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:41825 every 8 connection(s)
I20260808 15:48:50.571156 1951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:50.575347 2053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:50.579511 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 1951
I20260808 15:48:50.579797 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:50.580720 27987 external_mini_cluster.cc:1428] Setting key 8b148258a67d8ef54cfb399f85abd950
I20260808 15:48:50.592120 2053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a: Bootstrap starting.
I20260808 15:48:50.596309 2053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:50.597461 2053 log.cc:824] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:48:50 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:50.601447 2053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a: No bootstrap required, opened a new log
I20260808 15:48:50.607923 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:50.583478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.27.84.254:41825)
0808 15:48:50.584640 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:50.584658 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:50.585095 (+ 437us) server_negotiation.cc:407] Connection header received
0808 15:48:50.587122 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:50.587161 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:50.587780 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:50.588310 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:50.589651 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:50.591846 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:50.593216 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:50.594480 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:50.594752 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:50.599166 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:50.599178 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:50.599198 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:50.599239 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:50.603212 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:50.603898 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:50.603907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:50.603917 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:50.604143 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:50.604484 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:50.604491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:50.604495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:50.605311 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:50.605622 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:50.605959 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 15:48:50.606434 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":530,"threads_started":1}
I20260808 15:48:50.611471 2053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41825 } }
I20260808 15:48:50.611822 2053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:50.611904 2053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e2a9ba8a5e6491b9bbe516e80285f7a, State: Initialized, Role: FOLLOWER
I20260808 15:48:50.612418 2053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41825 } }
I20260808 15:48:50.612619 2053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:50.612732 2053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:50.612860 2053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:50.615232 2053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41825 } }
I20260808 15:48:50.615710 2053 leader_election.cc:302] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e2a9ba8a5e6491b9bbe516e80285f7a; no voters:
I20260808 15:48:50.616271 2053 leader_election.cc:288] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:50.616472 2061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:50.617736 2061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [term 1 LEADER]: Becoming Leader. State: Replica: 7e2a9ba8a5e6491b9bbe516e80285f7a, State: Running, Role: LEADER
I20260808 15:48:50.618263 2061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41825 } }
I20260808 15:48:50.618691 2053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:50.621387 2065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e2a9ba8a5e6491b9bbe516e80285f7a. Latest consensus state: current_term: 1 leader_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41825 } } }
I20260808 15:48:50.621794 2065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:50.621846 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e2a9ba8a5e6491b9bbe516e80285f7a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41825 } } }
I20260808 15:48:50.622288 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:50.625993 2076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:50.630735 2076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:50.640174 2076 catalog_manager.cc:1347] Generated new cluster ID: dfd5b5466ce94d3d8b1e4d5eab4a3398
I20260808 15:48:50.640308 2076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:50.656008 2076 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:50.657554 2076 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:50.667843 2076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a: Generated new TSK 0
I20260808 15:48:50.668763 2076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:48:50.739490 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41825
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:51.091423 2093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:51.091763 2093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:51.091934 2093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:51.099295 2093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:51.099449 2093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:51.099627 2093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:51.109803 2093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:51.111261 2093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:51.112684 2093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:51.126403 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:51.126582 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:51.126427 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:51.127513 2093 server_base.cc:1034] running on GCE node
I20260808 15:48:51.128153 2093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:51.129179 2093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:51.130452 2093 hybrid_clock.cc:648] HybridClock initialized: now 1786204131130383 us; error 56 us; skew 500 ppm
I20260808 15:48:51.130935 2093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:51 dist-test-slave-twwt krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:51.134320 2093 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:48:51.135588 2093 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:51.135754 2093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:51.136159 2093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:51.139427 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "4f654324767f4a88854f8c5e49ac057f"
format_stamp: "Formatted at 2026-08-08 15:48:51 on dist-test-slave-twwt"
server_key: "bdc5d9f012ffad1bc9bd738972c236c2"
server_key_iv: "d7d0e79beb7ee2afdb52859db1a5edb3"
server_key_version: "encryptionkey@0"
I20260808 15:48:51.140239 2093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "4f654324767f4a88854f8c5e49ac057f"
format_stamp: "Formatted at 2026-08-08 15:48:51 on dist-test-slave-twwt"
server_key: "bdc5d9f012ffad1bc9bd738972c236c2"
server_key_iv: "d7d0e79beb7ee2afdb52859db1a5edb3"
server_key_version: "encryptionkey@0"
I20260808 15:48:51.145493 2093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:48:51.149575 2113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:51.151314 2093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:51.151506 2093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "4f654324767f4a88854f8c5e49ac057f"
format_stamp: "Formatted at 2026-08-08 15:48:51 on dist-test-slave-twwt"
server_key: "bdc5d9f012ffad1bc9bd738972c236c2"
server_key_iv: "d7d0e79beb7ee2afdb52859db1a5edb3"
server_key_version: "encryptionkey@0"
I20260808 15:48:51.151752 2093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:51.181445 2093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:51.186003 2093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:51.186331 2093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:51.187484 2093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:51.188930 2093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:51.189059 2093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:51.189193 2093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:51.189281 2093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:51.223531 2093 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43569
I20260808 15:48:51.223562 2316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43569 every 8 connection(s)
I20260808 15:48:51.224475 2093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:51.227798 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 2093
I20260808 15:48:51.228040 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:51.228930 27987 external_mini_cluster.cc:1428] Setting key 97eff3da38d58731e39759a358e81ce8
Aug 08 15:48:51 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:51.248895 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:51.227121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:45763 (local address 127.27.84.254:41825)
0808 15:48:51.227623 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:51.227629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:51.228976 (+ 1347us) server_negotiation.cc:407] Connection header received
0808 15:48:51.231064 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:51.231068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:51.231129 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:51.231225 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:51.233429 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.234151 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.236726 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.237183 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.237286 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:51.241736 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:51.241746 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:51.241753 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:51.241800 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:51.245479 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:51.246205 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:51.246214 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:51.246221 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:51.246376 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:51.246855 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:51.246863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:51.246869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:51.247272 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:51.247432 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:51.248096 (+ 664us) server_negotiation.cc:299] Negotiation successful
0808 15:48:51.248379 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":250,"threads_started":1}
I20260808 15:48:51.249488 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:51.227769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41825 (local address 127.27.84.193:45763)
0808 15:48:51.228773 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:51.228829 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:51.230781 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:51.231500 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:51.231511 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:51.232335 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:51.233228 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:51.233258 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.234299 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.234307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:51.236558 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:51.236576 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.237435 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.237446 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:51.237776 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:51.238544 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:51.238580 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:51.241493 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:51.245640 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:51.245647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:51.245659 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:51.246068 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:51.246483 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:51.246486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:51.246488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:51.246714 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:51.247395 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:51.247401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:51.247706 (+ 305us) client_negotiation.cc:769] Sending connection context
0808 15:48:51.247926 (+ 220us) client_negotiation.cc:240] Negotiation successful
0808 15:48:51.248315 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":401,"threads_started":1}
I20260808 15:48:51.251204 2317 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41825
I20260808 15:48:51.251573 2317 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:51.252465 2317 heartbeater.cc:511] Master 127.27.84.254:41825 requested a full tablet report, sending...
I20260808 15:48:51.255244 1999 ts_manager.cc:194] Registered new tserver with Master: 4f654324767f4a88854f8c5e49ac057f (127.27.84.193:43569)
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
I20260808 15:48:51.257411 1999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:45763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:48:51.289469 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41825
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:51.632537 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:51.632891 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:51.633055 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:51.640965 2324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:51.641119 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:51.641314 2324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:51.651402 2324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:51.652865 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:51.654268 2324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:51.667615 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:51.667726 2332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:51.667701 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:51.668654 2324 server_base.cc:1034] running on GCE node
I20260808 15:48:51.669313 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:51.670251 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:51.671447 2324 hybrid_clock.cc:648] HybridClock initialized: now 1786204131671400 us; error 36 us; skew 500 ppm
I20260808 15:48:51.671756 2324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:51 dist-test-slave-twwt krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:51.675055 2324 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:48:51.676170 2324 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:51.676285 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:51.676616 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:51.679658 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "954aec0aef634c0c9f4f144327be1bf9"
format_stamp: "Formatted at 2026-08-08 15:48:51 on dist-test-slave-twwt"
server_key: "6d1957f06eb7aec65e8d3a1b0ebcc8b8"
server_key_iv: "55ea8996a59e0d45f9bf29e686432098"
server_key_version: "encryptionkey@0"
I20260808 15:48:51.680495 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "954aec0aef634c0c9f4f144327be1bf9"
format_stamp: "Formatted at 2026-08-08 15:48:51 on dist-test-slave-twwt"
server_key: "6d1957f06eb7aec65e8d3a1b0ebcc8b8"
server_key_iv: "55ea8996a59e0d45f9bf29e686432098"
server_key_version: "encryptionkey@0"
I20260808 15:48:51.685739 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260808 15:48:51.689229 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:51.690472 2324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:51.690697 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "954aec0aef634c0c9f4f144327be1bf9"
format_stamp: "Formatted at 2026-08-08 15:48:51 on dist-test-slave-twwt"
server_key: "6d1957f06eb7aec65e8d3a1b0ebcc8b8"
server_key_iv: "55ea8996a59e0d45f9bf29e686432098"
server_key_version: "encryptionkey@0"
I20260808 15:48:51.690940 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:51.710201 2324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:51.714644 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:51.714967 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:51.715999 2324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:51.717366 2324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:51.717501 2324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:51.717626 2324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:51.717710 2324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:51.753470 2324 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:37533
I20260808 15:48:51.753527 2519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:37533 every 8 connection(s)
I20260808 15:48:51.754362 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:51.764406 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 2324
I20260808 15:48:51.764626 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:51.765159 27987 external_mini_cluster.cc:1428] Setting key 47337dda449d84ec74a710312496e292
Aug 08 15:48:51 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:51.775873 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:51.757074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:48113 (local address 127.27.84.254:41825)
0808 15:48:51.757550 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:51.757556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:51.758629 (+ 1073us) server_negotiation.cc:407] Connection header received
0808 15:48:51.760744 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:51.760749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:51.760818 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:51.760937 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:51.762472 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.763432 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.765633 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.766162 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.766284 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:51.769990 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:51.769995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:51.769998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:51.770030 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:51.772680 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:51.773439 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:51.773449 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:51.773451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:51.773576 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:51.774003 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:51.774008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:51.774012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:51.774293 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:51.774452 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:51.775244 (+ 792us) server_negotiation.cc:299] Negotiation successful
0808 15:48:51.775474 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":232,"threads_started":1}
I20260808 15:48:51.776983 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:51.757731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41825 (local address 127.27.84.194:48113)
0808 15:48:51.758436 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:51.758476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:51.760474 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:51.761164 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:51.761172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:51.761727 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:51.762309 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:51.762328 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.763586 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.763590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:51.765489 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:51.765505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:51.766415 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:51.766423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:51.766716 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:51.767537 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:51.767561 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:51.769758 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:51.772838 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:51.772847 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:51.772864 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:51.773309 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:51.773690 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:51.773695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:51.773699 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:51.773893 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:51.774437 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:51.774446 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:51.774874 (+ 428us) client_negotiation.cc:769] Sending connection context
0808 15:48:51.775147 (+ 273us) client_negotiation.cc:240] Negotiation successful
0808 15:48:51.775727 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":277,"threads_started":1}
I20260808 15:48:51.778543 2520 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41825
I20260808 15:48:51.778942 2520 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:51.779794 2520 heartbeater.cc:511] Master 127.27.84.254:41825 requested a full tablet report, sending...
I20260808 15:48:51.781613 1999 ts_manager.cc:194] Registered new tserver with Master: 954aec0aef634c0c9f4f144327be1bf9 (127.27.84.194:37533)
I20260808 15:48:51.782994 1999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:48113
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:48:51.824236 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41825
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:48:52.178793 2527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:52.179087 2527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:52.179188 2527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:52.186560 2527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:52.186733 2527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:52.186873 2527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:52.197140 2527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:52.198479 2527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:52.199941 2527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:52.213413 2533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:52.213382 2536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:52.213455 2534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:52.214676 2527 server_base.cc:1034] running on GCE node
I20260808 15:48:52.215291 2527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:52.216331 2527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:52.217538 2527 hybrid_clock.cc:648] HybridClock initialized: now 1786204132217491 us; error 39 us; skew 500 ppm
I20260808 15:48:52.217880 2527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:48:52 dist-test-slave-twwt krb5kdc[1925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204132, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:48:52.221289 2527 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:48:52.222528 2527 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:52.222673 2527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:52.223131 2527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:52.226184 2527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "2a09100ceb8c4894b9039e2cdc6bf71b"
format_stamp: "Formatted at 2026-08-08 15:48:52 on dist-test-slave-twwt"
server_key: "7ffbb8ecc918a84bf99c86a851e7565f"
server_key_iv: "f2e76808f26cd2c3be003d0ee4a8c27c"
server_key_version: "encryptionkey@0"
I20260808 15:48:52.227065 2527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "2a09100ceb8c4894b9039e2cdc6bf71b"
format_stamp: "Formatted at 2026-08-08 15:48:52 on dist-test-slave-twwt"
server_key: "7ffbb8ecc918a84bf99c86a851e7565f"
server_key_iv: "f2e76808f26cd2c3be003d0ee4a8c27c"
server_key_version: "encryptionkey@0"
I20260808 15:48:52.232059 2527 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:48:52.235493 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:52.236806 2527 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 15:48:52.236994 2527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "2a09100ceb8c4894b9039e2cdc6bf71b"
format_stamp: "Formatted at 2026-08-08 15:48:52 on dist-test-slave-twwt"
server_key: "7ffbb8ecc918a84bf99c86a851e7565f"
server_key_iv: "f2e76808f26cd2c3be003d0ee4a8c27c"
server_key_version: "encryptionkey@0"
I20260808 15:48:52.237206 2527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:52.260877 2317 heartbeater.cc:503] Master 127.27.84.254:41825 was elected leader, sending a full tablet report...
I20260808 15:48:52.263022 2527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:52.267028 2527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:52.267256 2527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:52.268148 2527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:52.269450 2527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:52.269538 2527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:52.269615 2527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:52.269670 2527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:52.303568 2527 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:43443
I20260808 15:48:52.303671 2730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:43443 every 8 connection(s)
I20260808 15:48:52.304450 2527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:52.305526 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 2527
I20260808 15:48:52.305739 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:52.306313 27987 external_mini_cluster.cc:1428] Setting key 55d192c6e3328261d3b6ac827bcd7c75
Aug 08 15:48:52 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204132, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:48:52.325659 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.307496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51189 (local address 127.27.84.254:41825)
0808 15:48:52.308011 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.308016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.309235 (+ 1219us) server_negotiation.cc:407] Connection header received
0808 15:48:52.311286 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.311290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.311361 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.311458 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:52.312934 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.313732 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.315327 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.315753 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.315848 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.319471 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:52.319476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:52.319479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:52.319509 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:52.322857 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.323593 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.323597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.323600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.323745 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.324147 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.324150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.324152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.324432 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:52.324540 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.325190 (+ 650us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.325365 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":244,"threads_started":1}
I20260808 15:48:52.326772 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.308158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41825 (local address 127.27.84.195:51189)
0808 15:48:52.309052 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:52.309118 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:52.311049 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:52.311654 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:52.311662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:52.312190 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:52.312779 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.312797 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.313874 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.313878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.315196 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.315208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.316012 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.316018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.316228 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.316953 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:48:52.316976 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:52.319261 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:52.323001 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:52.323008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:52.323022 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:52.323482 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:52.323856 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:48:52.323859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:48:52.323862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:48:52.324049 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:48:52.324534 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:52.324540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:52.324859 (+ 319us) client_negotiation.cc:769] Sending connection context
0808 15:48:52.325090 (+ 231us) client_negotiation.cc:240] Negotiation successful
0808 15:48:52.325515 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":342,"threads_started":1}
I20260808 15:48:52.328223 2731 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41825
I20260808 15:48:52.328545 2731 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:52.329244 2731 heartbeater.cc:511] Master 127.27.84.254:41825 requested a full tablet report, sending...
I20260808 15:48:52.330928 1999 ts_manager.cc:194] Registered new tserver with Master: 2a09100ceb8c4894b9039e2cdc6bf71b (127.27.84.195:43443)
I20260808 15:48:52.331781 1999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:51189
I20260808 15:48:52.334971 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:48:52.352479 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.339974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.27.84.254:41825)
0808 15:48:52.340164 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.340170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.340806 (+ 636us) server_negotiation.cc:407] Connection header received
0808 15:48:52.340946 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.340949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.341012 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.341099 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:52.342594 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.343337 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.344613 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.345017 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.345138 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.347498 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:52.347503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:52.347506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:52.347537 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:52.350219 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.350900 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.350904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.350907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.351022 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.351350 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.351353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.351355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.351654 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:52.351872 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.352089 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.352257 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:48:52.356660 1999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:52.363826 1999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:52.392839 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.376786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.27.84.195:43443)
0808 15:48:52.385230 (+ 8444us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.385237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.385270 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:48:52.385384 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.385392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.385596 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.385790 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:52.386432 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.388001 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.390121 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.391566 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.391730 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.391977 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.392262 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.392443 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8257,"thread_start_us":8100,"threads_started":1}
I20260808 15:48:52.392926 2751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.376465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:43443 (local address 127.0.0.1:52416)
0808 15:48:52.377690 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:52.377762 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:52.377993 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:52.385782 (+ 7789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:52.385786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:52.385813 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:52.386214 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.386228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.388149 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.388154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.389959 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.389972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.391769 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.391775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.391983 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.392054 (+ 71us) client_negotiation.cc:769] Sending connection context
0808 15:48:52.392198 (+ 144us) client_negotiation.cc:240] Negotiation successful
0808 15:48:52.392400 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":82,"thread_start_us":8070,"threads_started":1}
I20260808 15:48:52.393326 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.376807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37533 (local address 127.0.0.1:60076)
0808 15:48:52.380159 (+ 3352us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:52.380187 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:52.380321 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:52.385231 (+ 4910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:52.385241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:52.385275 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:52.385647 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.385660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.387675 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.387680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.390260 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.390275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.392110 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.392115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.392300 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.392333 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:48:52.392400 (+ 67us) client_negotiation.cc:240] Negotiation successful
0808 15:48:52.392515 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3198,"mutex_wait_us":26,"thread_start_us":1549,"threads_started":1}
I20260808 15:48:52.393925 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.377507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.27.84.194:37533)
0808 15:48:52.384418 (+ 6911us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.384430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.384469 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:48:52.384597 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.384607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.384921 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.385205 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:52.385856 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.387542 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.390432 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.391869 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.393097 (+ 1228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.393322 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.393405 (+ 83us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.393585 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6707,"thread_start_us":6549,"threads_started":1}
I20260808 15:48:52.394318 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.379875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.27.84.193:43569)
0808 15:48:52.383683 (+ 3808us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.383704 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.385016 (+ 1312us) server_negotiation.cc:407] Connection header received
0808 15:48:52.385341 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.385351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.385585 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.385804 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:52.386819 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.388131 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.390773 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.392265 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.392402 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.392619 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.393792 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.393972 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3604,"thread_start_us":3435,"threads_started":1}
I20260808 15:48:52.394368 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.379506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43569 (local address 127.0.0.1:47430)
0808 15:48:52.384873 (+ 5367us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:52.384902 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:52.385129 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:52.386127 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:52.386131 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:52.386155 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:52.386579 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.386591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.388278 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.388288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.390542 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.390551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.392785 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.392791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.393576 (+ 785us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.393615 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:48:52.393718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:48:52.393846 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5197,"mutex_wait_us":35,"thread_start_us":307,"threads_started":1}
I20260808 15:48:52.397547 2416 tablet_service.cc:1467] Processing CreateTablet for tablet 37747e72616647718f9ca71e46d10afe (DEFAULT_TABLE table=test-table [id=c6fe217da8c6443bace2116a9f33508a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:52.397604 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 37747e72616647718f9ca71e46d10afe (DEFAULT_TABLE table=test-table [id=c6fe217da8c6443bace2116a9f33508a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:52.398141 2203 tablet_service.cc:1467] Processing CreateTablet for tablet 37747e72616647718f9ca71e46d10afe (DEFAULT_TABLE table=test-table [id=c6fe217da8c6443bace2116a9f33508a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:52.398912 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37747e72616647718f9ca71e46d10afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:52.399096 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37747e72616647718f9ca71e46d10afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:52.399500 2203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37747e72616647718f9ca71e46d10afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:52.410980 2759 tablet_bootstrap.cc:492] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Bootstrap starting.
I20260808 15:48:52.414162 2759 tablet_bootstrap.cc:654] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:52.415050 2760 tablet_bootstrap.cc:492] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Bootstrap starting.
I20260808 15:48:52.415570 2759 log.cc:824] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:52.415977 2761 tablet_bootstrap.cc:492] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Bootstrap starting.
I20260808 15:48:52.419442 2761 tablet_bootstrap.cc:654] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:52.419494 2759 tablet_bootstrap.cc:492] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: No bootstrap required, opened a new log
I20260808 15:48:52.419555 2760 tablet_bootstrap.cc:654] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:52.419780 2759 ts_tablet_manager.cc:1397] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260808 15:48:52.420619 2761 log.cc:824] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:52.420956 2760 log.cc:824] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:52.423820 2760 tablet_bootstrap.cc:492] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: No bootstrap required, opened a new log
I20260808 15:48:52.423956 2761 tablet_bootstrap.cc:492] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: No bootstrap required, opened a new log
I20260808 15:48:52.424132 2760 ts_tablet_manager.cc:1397] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260808 15:48:52.424242 2761 ts_tablet_manager.cc:1397] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:48:52.430694 2759 raft_consensus.cc:348] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.431174 2759 raft_consensus.cc:374] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:52.431282 2759 raft_consensus.cc:729] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954aec0aef634c0c9f4f144327be1bf9, State: Initialized, Role: FOLLOWER
I20260808 15:48:52.431823 2760 raft_consensus.cc:348] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.431965 2759 consensus_queue.cc:260] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.432156 2760 raft_consensus.cc:374] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:52.432301 2760 raft_consensus.cc:729] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a09100ceb8c4894b9039e2cdc6bf71b, State: Initialized, Role: FOLLOWER
I20260808 15:48:52.433204 2520 heartbeater.cc:503] Master 127.27.84.254:41825 was elected leader, sending a full tablet report...
I20260808 15:48:52.433055 2760 consensus_queue.cc:260] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.433861 2759 ts_tablet_manager.cc:1428] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260808 15:48:52.434664 2761 raft_consensus.cc:348] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.435125 2761 raft_consensus.cc:374] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:52.435227 2761 raft_consensus.cc:729] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f654324767f4a88854f8c5e49ac057f, State: Initialized, Role: FOLLOWER
I20260808 15:48:52.435315 2731 heartbeater.cc:503] Master 127.27.84.254:41825 was elected leader, sending a full tablet report...
I20260808 15:48:52.435956 2761 consensus_queue.cc:260] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.437350 2760 ts_tablet_manager.cc:1428] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260808 15:48:52.438858 2761 ts_tablet_manager.cc:1428] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
I20260808 15:48:52.444445 2775 raft_consensus.cc:482] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:52.444695 2775 raft_consensus.cc:504] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.446168 2775 leader_election.cc:288] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 954aec0aef634c0c9f4f144327be1bf9 (127.27.84.194:37533), 4f654324767f4a88854f8c5e49ac057f (127.27.84.193:43569)
I20260808 15:48:52.452404 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.446552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:54859 (local address 127.27.84.194:37533)
0808 15:48:52.446753 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.446757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.446882 (+ 125us) server_negotiation.cc:407] Connection header received
0808 15:48:52.447141 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.447147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.447236 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.447347 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:52.447868 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.448589 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.450584 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.451450 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.451622 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.451698 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.451983 (+ 285us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.452125 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:48:52.452416 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.446561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37533 (local address 127.27.84.195:54859)
0808 15:48:52.446786 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:52.446809 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:52.446944 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:52.447383 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:52.447391 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:52.447431 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:52.447710 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.447723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.448734 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.448745 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.450413 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.450430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.451622 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.451629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.451820 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.451845 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:48:52.451954 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:48:52.452124 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:48:52.453702 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.447182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:33041 (local address 127.27.84.193:43569)
0808 15:48:52.447359 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.447365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.447966 (+ 601us) server_negotiation.cc:407] Connection header received
0808 15:48:52.448133 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.448138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.448203 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.448295 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:48:52.448876 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.449584 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.451533 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.452287 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.452477 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.452547 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.453312 (+ 765us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.453453 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260808 15:48:52.453531 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37747e72616647718f9ca71e46d10afe" candidate_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954aec0aef634c0c9f4f144327be1bf9" is_pre_election: true
I20260808 15:48:52.453755 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.447010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43569 (local address 127.27.84.195:33041)
0808 15:48:52.447866 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:52.447882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:52.448014 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:52.448311 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:52.448315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:52.448339 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:48:52.448683 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.448700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.449730 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.449738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.451380 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:52.451393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.452500 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.452975 (+ 475us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:52.453155 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.453180 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:48:52.453284 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:48:52.453416 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":496,"threads_started":1}
I20260808 15:48:52.454134 2444 raft_consensus.cc:2454] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a09100ceb8c4894b9039e2cdc6bf71b in term 0.
I20260808 15:48:52.454743 2237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37747e72616647718f9ca71e46d10afe" candidate_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f654324767f4a88854f8c5e49ac057f" is_pre_election: true
I20260808 15:48:52.454874 2558 leader_election.cc:302] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a09100ceb8c4894b9039e2cdc6bf71b, 954aec0aef634c0c9f4f144327be1bf9; no voters:
I20260808 15:48:52.455267 2775 raft_consensus.cc:2781] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:52.455256 2237 raft_consensus.cc:2454] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a09100ceb8c4894b9039e2cdc6bf71b in term 0.
I20260808 15:48:52.455428 2775 raft_consensus.cc:482] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:52.455545 2775 raft_consensus.cc:3037] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:52.458403 2775 raft_consensus.cc:504] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.459142 2775 leader_election.cc:288] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [CANDIDATE]: Term 1 election: Requested vote from peers 954aec0aef634c0c9f4f144327be1bf9 (127.27.84.194:37533), 4f654324767f4a88854f8c5e49ac057f (127.27.84.193:43569)
I20260808 15:48:52.459535 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37747e72616647718f9ca71e46d10afe" candidate_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954aec0aef634c0c9f4f144327be1bf9"
I20260808 15:48:52.459635 2237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37747e72616647718f9ca71e46d10afe" candidate_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f654324767f4a88854f8c5e49ac057f"
I20260808 15:48:52.459750 2444 raft_consensus.cc:3037] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:52.459880 2237 raft_consensus.cc:3037] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:52.462167 2444 raft_consensus.cc:2454] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a09100ceb8c4894b9039e2cdc6bf71b in term 1.
I20260808 15:48:52.462179 2237 raft_consensus.cc:2454] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a09100ceb8c4894b9039e2cdc6bf71b in term 1.
I20260808 15:48:52.462589 2558 leader_election.cc:302] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a09100ceb8c4894b9039e2cdc6bf71b, 954aec0aef634c0c9f4f144327be1bf9; no voters:
I20260808 15:48:52.462965 2775 raft_consensus.cc:2781] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:52.463425 2775 raft_consensus.cc:686] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 1 LEADER]: Becoming Leader. State: Replica: 2a09100ceb8c4894b9039e2cdc6bf71b, State: Running, Role: LEADER
I20260808 15:48:52.463956 2775 consensus_queue.cc:237] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } }
I20260808 15:48:52.468904 1999 catalog_manager.cc:5495] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a09100ceb8c4894b9039e2cdc6bf71b (127.27.84.195). New cstate: current_term: 1 leader_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a09100ceb8c4894b9039e2cdc6bf71b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 43443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 } health_report { overall_health: UNKNOWN } } }
W20260808 15:48:52.476907 2318 tablet.cc:2369] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:52.489552 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.484507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.27.84.195:43443)
0808 15:48:52.484732 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.484737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.484755 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:52.484935 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.484939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.485026 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.485146 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:48:52.485484 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.486232 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.487580 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.487980 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.488076 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.488488 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:48:52.489002 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:48:52.489109 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.489200 (+ 91us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.489262 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:48:52.497651 2444 raft_consensus.cc:1261] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 1 FOLLOWER]: Refusing update from remote peer 2a09100ceb8c4894b9039e2cdc6bf71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:52.497653 2237 raft_consensus.cc:1261] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 1 FOLLOWER]: Refusing update from remote peer 2a09100ceb8c4894b9039e2cdc6bf71b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:52.498595 2778 consensus_queue.cc:1035] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f654324767f4a88854f8c5e49ac057f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:52.499002 2775 consensus_queue.cc:1035] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [LEADER]: Connected to new peer: Peer: permanent_uuid: "954aec0aef634c0c9f4f144327be1bf9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260808 15:48:52.506300 2521 tablet.cc:2369] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:52.524876 1997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:52.525214 1997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:52.529168 1997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e2a9ba8a5e6491b9bbe516e80285f7a: Sending DeleteTablet for 3 replicas of tablet 37747e72616647718f9ca71e46d10afe
I20260808 15:48:52.530401 2629 tablet_service.cc:1514] Processing DeleteTablet for tablet 37747e72616647718f9ca71e46d10afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:52 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:52416
I20260808 15:48:52.530630 2416 tablet_service.cc:1514] Processing DeleteTablet for tablet 37747e72616647718f9ca71e46d10afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:52 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:60076
I20260808 15:48:52.530732 2203 tablet_service.cc:1514] Processing DeleteTablet for tablet 37747e72616647718f9ca71e46d10afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:52 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:47430
I20260808 15:48:52.533023 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 2093
I20260808 15:48:52.545763 2803 tablet_replica.cc:330] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: stopping tablet replica
I20260808 15:48:52.546296 2803 raft_consensus.cc:2229] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:52.547197 2803 raft_consensus.cc:2258] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:52.549521 2803 ts_tablet_manager.cc:1905] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:52.557453 2803 ts_tablet_manager.cc:1918] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:52.557653 2803 log.cc:1196] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/37747e72616647718f9ca71e46d10afe
I20260808 15:48:52.558281 2803 ts_tablet_manager.cc:1939] T 37747e72616647718f9ca71e46d10afe P 2a09100ceb8c4894b9039e2cdc6bf71b: Deleting consensus metadata
I20260808 15:48:52.560432 1981 catalog_manager.cc:4841] TS 2a09100ceb8c4894b9039e2cdc6bf71b (127.27.84.195:43443): tablet 37747e72616647718f9ca71e46d10afe (table test-table [id=c6fe217da8c6443bace2116a9f33508a]) successfully deleted
I20260808 15:48:52.563374 2809 tablet_replica.cc:330] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: stopping tablet replica
I20260808 15:48:52.564025 2809 raft_consensus.cc:2229] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:52.564723 2809 raft_consensus.cc:2258] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:52.567490 2805 tablet_replica.cc:330] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: stopping tablet replica
I20260808 15:48:52.567862 2809 ts_tablet_manager.cc:1905] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:52.568147 2805 raft_consensus.cc:2229] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:52.568738 2805 raft_consensus.cc:2258] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:52.571128 2805 ts_tablet_manager.cc:1905] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:52.575281 2809 ts_tablet_manager.cc:1918] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:52.575498 2809 log.cc:1196] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/37747e72616647718f9ca71e46d10afe
I20260808 15:48:52.576054 2809 ts_tablet_manager.cc:1939] T 37747e72616647718f9ca71e46d10afe P 4f654324767f4a88854f8c5e49ac057f: Deleting consensus metadata
I20260808 15:48:52.578114 1984 catalog_manager.cc:4841] TS 4f654324767f4a88854f8c5e49ac057f (127.27.84.193:43569): tablet 37747e72616647718f9ca71e46d10afe (table test-table [id=c6fe217da8c6443bace2116a9f33508a]) successfully deleted
I20260808 15:48:52.578464 2805 ts_tablet_manager.cc:1918] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:52.578687 2805 log.cc:1196] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/37747e72616647718f9ca71e46d10afe
I20260808 15:48:52.579181 2805 ts_tablet_manager.cc:1939] T 37747e72616647718f9ca71e46d10afe P 954aec0aef634c0c9f4f144327be1bf9: Deleting consensus metadata
Aug 08 15:48:52 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:48:52.581146 1984 catalog_manager.cc:4841] TS 954aec0aef634c0c9f4f144327be1bf9 (127.27.84.194:37533): tablet 37747e72616647718f9ca71e46d10afe (table test-table [id=c6fe217da8c6443bace2116a9f33508a]) successfully deleted
I20260808 15:48:52.588627 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.534948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.27.84.193:43569)
0808 15:48:52.535195 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.535204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.559297 (+ 24093us) server_negotiation.cc:407] Connection header received
0808 15:48:52.559941 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.559950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.560075 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.560169 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:52.562602 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.564029 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.566025 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.566742 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.566853 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.582358 (+ 15505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:52.582373 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:52.582389 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:52.582439 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:52.585466 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.586271 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.586280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.586283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.586438 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.586915 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.586926 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.586928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.587338 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:52.587680 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.587985 (+ 305us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.588219 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:48:52.589546 2311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:52.681945 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 2093
I20260808 15:48:52.710709 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 2324
Aug 08 15:48:52 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:48:52.729640 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.711635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60088 (local address 127.27.84.194:37533)
0808 15:48:52.711803 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.711808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.711826 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:52.711984 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.711988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.712051 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.712136 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:52.713849 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.714849 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.716692 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.717085 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.717186 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.722318 (+ 5132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:52.722343 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:52.722373 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:52.722432 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:52.726301 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.727222 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.727232 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.727236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.727416 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.727875 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.727880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.727884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.728374 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:52.728681 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.729002 (+ 321us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.729245 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260808 15:48:52.730489 2512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:52.827208 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 2324
I20260808 15:48:52.866158 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 2527
Aug 08 15:48:52 dist-test-slave-twwt krb5kdc[1925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:48:52.887854 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:52.867060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.27.84.195:43443)
0808 15:48:52.867275 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:52.867292 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:52.867316 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:48:52.867448 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:52.867454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:52.867542 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:52.867662 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:52.868982 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.870250 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.871785 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:52.872488 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:52.872626 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:52.880507 (+ 7881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:52.880530 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:52.880564 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:48:52.880618 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:52.884805 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.885472 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.885482 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.885485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.885682 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:48:52.886054 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:48:52.886059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:48:52.886061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:48:52.886593 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:52.886926 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:52.887136 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 15:48:52.887390 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:48:52.888687 2725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:53.031296 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 2527
I20260808 15:48:53.068005 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 1951
I20260808 15:48:53.069203 2044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:53.191008 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 1951
2026-08-08T15:48:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4633 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-08T15:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:53Z Disabled control of system clock
I20260808 15:48:53.278112 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43623
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 15:48:53.622335 2821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:53.622680 2821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:53.622850 2821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:53.630206 2821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:53.630316 2821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:53.630381 2821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:53.630468 2821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:53.630540 2821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:48:53.640125 2821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:53.641537 2821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:53.643527 2821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:53.655809 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:53.655905 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:53.655844 2836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:53.656704 2821 server_base.cc:1034] running on GCE node
I20260808 15:48:53.657671 2821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:53.659237 2821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:53.660493 2821 hybrid_clock.cc:648] HybridClock initialized: now 1786204133660430 us; error 48 us; skew 500 ppm
I20260808 15:48:53.660907 2821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:53.663290 2821 webserver.cc:460] Webserver started at http://127.27.84.254:33187/ using document root <none> and password file <none>
I20260808 15:48:53.664060 2821 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:53.664196 2821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:53.664531 2821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:53.667582 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "529a04ce6eab4aa1983a3d536c7b14b7"
format_stamp: "Formatted at 2026-08-08 15:48:53 on dist-test-slave-twwt"
server_key: "aa2a5f10662b899fbd682f5e5854458c"
server_key_iv: "a1bebd5e3a8726d7384de62a9b4e5704"
server_key_version: "encryptionkey@0"
I20260808 15:48:53.668437 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "529a04ce6eab4aa1983a3d536c7b14b7"
format_stamp: "Formatted at 2026-08-08 15:48:53 on dist-test-slave-twwt"
server_key: "aa2a5f10662b899fbd682f5e5854458c"
server_key_iv: "a1bebd5e3a8726d7384de62a9b4e5704"
server_key_version: "encryptionkey@0"
I20260808 15:48:53.673771 2821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:48:53.677733 2846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:53.679203 2821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:53.679385 2821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "529a04ce6eab4aa1983a3d536c7b14b7"
format_stamp: "Formatted at 2026-08-08 15:48:53 on dist-test-slave-twwt"
server_key: "aa2a5f10662b899fbd682f5e5854458c"
server_key_iv: "a1bebd5e3a8726d7384de62a9b4e5704"
server_key_version: "encryptionkey@0"
I20260808 15:48:53.679644 2821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:53.685572 2821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:53.686395 2821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:53.686764 2821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:53.705722 2821 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43623
I20260808 15:48:53.705740 2927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43623 every 8 connection(s)
I20260808 15:48:53.707267 2821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:53.710632 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 2821
I20260808 15:48:53.710844 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:53.711732 27987 external_mini_cluster.cc:1428] Setting key 8000753a4c01a3b597420574727e6fa6
I20260808 15:48:53.711872 2929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:53.720790 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:53.713895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51824 (local address 127.27.84.254:43623)
0808 15:48:53.714995 (+ 1100us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:53.715007 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:53.715053 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:48:53.716696 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:53.716721 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:53.717228 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:53.717681 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:53.718926 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:53.718932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:53.718946 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:53.718964 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:53.719040 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:53.719061 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:53.719358 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:53.719554 (+ 196us) server_negotiation.cc:299] Negotiation successful
0808 15:48:53.719818 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":322,"threads_started":1}
I20260808 15:48:53.726526 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7: Bootstrap starting.
I20260808 15:48:53.729868 2929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:53.730966 2929 log.cc:824] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:53.733771 2929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7: No bootstrap required, opened a new log
I20260808 15:48:53.741366 2929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43623 } }
I20260808 15:48:53.741762 2929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:53.741884 2929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 529a04ce6eab4aa1983a3d536c7b14b7, State: Initialized, Role: FOLLOWER
I20260808 15:48:53.742450 2929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43623 } }
I20260808 15:48:53.742717 2929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:53.742865 2929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:53.743038 2929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:53.745437 2929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43623 } }
I20260808 15:48:53.745947 2929 leader_election.cc:302] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 529a04ce6eab4aa1983a3d536c7b14b7; no voters:
I20260808 15:48:53.746688 2929 leader_election.cc:288] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:53.746877 2941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:53.747829 2941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [term 1 LEADER]: Becoming Leader. State: Replica: 529a04ce6eab4aa1983a3d536c7b14b7, State: Running, Role: LEADER
I20260808 15:48:53.749336 2941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43623 } }
I20260808 15:48:53.749758 2929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:53.752483 2946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43623 } } }
I20260808 15:48:53.753216 2946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:53.753078 2945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 529a04ce6eab4aa1983a3d536c7b14b7. Latest consensus state: current_term: 1 leader_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "529a04ce6eab4aa1983a3d536c7b14b7" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43623 } } }
I20260808 15:48:53.753510 2945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:53.758188 2951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:53.763782 2951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:53.773200 2951 catalog_manager.cc:1347] Generated new cluster ID: 84179bfc17684b3abc691fc0fad1dcc6
I20260808 15:48:53.773355 2951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:53.791254 2951 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:53.791776 2951 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:53.804878 2951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7: Generated new TSK 0
I20260808 15:48:53.805980 2951 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:48:53.822680 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43623
--builtin_ntp_servers=127.27.84.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260808 15:48:54.177865 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:54.178174 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:54.178270 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:54.185881 2976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:54.185999 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:54.186111 2976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:48:54.195634 2976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:54.196954 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:54.198844 2976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:54.212594 2983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:54.212663 2987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:54.212632 2989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:54.213788 2976 server_base.cc:1034] running on GCE node
I20260808 15:48:54.214525 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:54.215627 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:54.216868 2976 hybrid_clock.cc:648] HybridClock initialized: now 1786204134216810 us; error 43 us; skew 500 ppm
I20260808 15:48:54.217221 2976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:54.219614 2976 webserver.cc:460] Webserver started at http://127.27.84.193:40843/ using document root <none> and password file <none>
I20260808 15:48:54.220484 2976 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:54.220580 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:54.220955 2976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:54.224006 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "269a26ae1d79435384f3335d462e4c04"
format_stamp: "Formatted at 2026-08-08 15:48:54 on dist-test-slave-twwt"
server_key: "21adf892f5d7038eca9fdd9ec4bcba4d"
server_key_iv: "b159ec5165e1a22724cbe3b0c0fa73f4"
server_key_version: "encryptionkey@0"
I20260808 15:48:54.224822 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "269a26ae1d79435384f3335d462e4c04"
format_stamp: "Formatted at 2026-08-08 15:48:54 on dist-test-slave-twwt"
server_key: "21adf892f5d7038eca9fdd9ec4bcba4d"
server_key_iv: "b159ec5165e1a22724cbe3b0c0fa73f4"
server_key_version: "encryptionkey@0"
I20260808 15:48:54.230202 2976 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:48:54.233882 2999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:54.235275 2976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:54.235457 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "269a26ae1d79435384f3335d462e4c04"
format_stamp: "Formatted at 2026-08-08 15:48:54 on dist-test-slave-twwt"
server_key: "21adf892f5d7038eca9fdd9ec4bcba4d"
server_key_iv: "b159ec5165e1a22724cbe3b0c0fa73f4"
server_key_version: "encryptionkey@0"
I20260808 15:48:54.235656 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:54.241869 2976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:54.242813 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:54.243090 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:54.244045 2976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:54.245489 2976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:54.245599 2976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:54.245723 2976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:54.245816 2976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:54.280494 2976 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:42415
I20260808 15:48:54.280568 3188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:42415 every 8 connection(s)
I20260808 15:48:54.281898 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:48:54.290584 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:54.284556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55309 (local address 127.27.84.254:43623)
0808 15:48:54.285029 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:54.285036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:54.286452 (+ 1416us) server_negotiation.cc:407] Connection header received
0808 15:48:54.288381 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:54.288385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:54.288440 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:54.288539 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:54.289930 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:54.289934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:54.289937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:54.289952 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:54.289989 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:54.290000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:54.290097 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:54.290290 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 15:48:54.290343 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":215,"threads_started":1}
I20260808 15:48:54.291640 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 2976
I20260808 15:48:54.291868 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:48:54.291960 3192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:54.285271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43623 (local address 127.27.84.193:55309)
0808 15:48:54.286292 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:54.286331 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:54.288096 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:54.288765 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:54.288774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:54.288871 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:54.289632 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:54.289635 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:54.289677 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:54.289775 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:54.290088 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:54.290097 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:54.290103 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 15:48:54.290352 (+ 249us) client_negotiation.cc:240] Negotiation successful
0808 15:48:54.290560 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":303,"threads_started":1}
I20260808 15:48:54.292868 27987 external_mini_cluster.cc:1428] Setting key 0b87d2b8dffd29a4e0b5f7b4ee969067
I20260808 15:48:54.293587 3189 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43623
I20260808 15:48:54.293924 3189 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:54.294557 3189 heartbeater.cc:511] Master 127.27.84.254:43623 requested a full tablet report, sending...
I20260808 15:48:54.296103 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43623
--builtin_ntp_servers=127.27.84.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 15:48:54.296882 2875 ts_manager.cc:194] Registered new tserver with Master: 269a26ae1d79435384f3335d462e4c04 (127.27.84.193:42415)
W20260808 15:48:54.641630 3193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:54.641974 3193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:54.642144 3193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:54.649361 3193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:54.649505 3193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:54.649684 3193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:48:54.658926 3193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:54.660333 3193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:54.662285 3193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:54.675818 3206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:54.675923 3202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:54.675869 3200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:54.680019 3193 server_base.cc:1034] running on GCE node
I20260808 15:48:54.680661 3193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:54.681658 3193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:54.682845 3193 hybrid_clock.cc:648] HybridClock initialized: now 1786204134682795 us; error 50 us; skew 500 ppm
I20260808 15:48:54.683236 3193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:54.685497 3193 webserver.cc:460] Webserver started at http://127.27.84.194:39865/ using document root <none> and password file <none>
I20260808 15:48:54.686204 3193 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:54.686316 3193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:54.686728 3193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:54.689682 3193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "1236070f1c154cbc9726a06528e51197"
format_stamp: "Formatted at 2026-08-08 15:48:54 on dist-test-slave-twwt"
server_key: "55970430812346ab68bb5ab3a33b06b5"
server_key_iv: "176556b395f010d777bd098a54af0a50"
server_key_version: "encryptionkey@0"
I20260808 15:48:54.690526 3193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "1236070f1c154cbc9726a06528e51197"
format_stamp: "Formatted at 2026-08-08 15:48:54 on dist-test-slave-twwt"
server_key: "55970430812346ab68bb5ab3a33b06b5"
server_key_iv: "176556b395f010d777bd098a54af0a50"
server_key_version: "encryptionkey@0"
I20260808 15:48:54.695997 3193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:48:54.699429 3214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:54.700677 3193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:54.700858 3193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "1236070f1c154cbc9726a06528e51197"
format_stamp: "Formatted at 2026-08-08 15:48:54 on dist-test-slave-twwt"
server_key: "55970430812346ab68bb5ab3a33b06b5"
server_key_iv: "176556b395f010d777bd098a54af0a50"
server_key_version: "encryptionkey@0"
I20260808 15:48:54.701082 3193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:54.706996 3193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:54.707795 3193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:54.708074 3193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:54.709026 3193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:54.710460 3193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:54.710580 3193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:54.710728 3193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:54.710816 3193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:54.745080 3193 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:46153
I20260808 15:48:54.745137 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:46153 every 8 connection(s)
I20260808 15:48:54.746470 3193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:48:54.753113 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 3193
I20260808 15:48:54.753316 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:48:54.753813 27987 external_mini_cluster.cc:1428] Setting key 7fbd2e1aab096c814291709989112c9f
I20260808 15:48:54.756522 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:54.749198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:43697 (local address 127.27.84.254:43623)
0808 15:48:54.749373 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:54.749378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:54.751752 (+ 2374us) server_negotiation.cc:407] Connection header received
0808 15:48:54.753912 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:54.753920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:54.754007 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:54.754134 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:54.755733 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:54.755742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:54.755748 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:54.755768 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:54.755813 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:54.755831 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:54.755953 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:54.756144 (+ 191us) server_negotiation.cc:299] Negotiation successful
0808 15:48:54.756220 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:48:54.757076 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43623
--builtin_ntp_servers=127.27.84.212:42799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260808 15:48:54.757506 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:54.749727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43623 (local address 127.27.84.194:43697)
0808 15:48:54.751120 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:54.751168 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:54.753605 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:54.754430 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:54.754439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:54.754546 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:54.755433 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:54.755437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:54.755491 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:54.755587 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:54.755929 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:54.755938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:54.755945 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:48:54.756231 (+ 286us) client_negotiation.cc:240] Negotiation successful
0808 15:48:54.756524 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":467,"threads_started":1}
I20260808 15:48:54.758978 3408 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43623
I20260808 15:48:54.759337 3408 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:54.760108 3408 heartbeater.cc:511] Master 127.27.84.254:43623 requested a full tablet report, sending...
I20260808 15:48:54.761826 2875 ts_manager.cc:194] Registered new tserver with Master: 1236070f1c154cbc9726a06528e51197 (127.27.84.194:46153)
W20260808 15:48:55.111224 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:55.111508 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:55.111675 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:55.118999 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:55.119100 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:55.119282 3412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:48:55.128434 3412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:55.129803 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:55.131688 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:55.145934 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:55.145988 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:55.145965 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:55.146891 3412 server_base.cc:1034] running on GCE node
I20260808 15:48:55.147595 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:55.148623 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:55.149873 3412 hybrid_clock.cc:648] HybridClock initialized: now 1786204135149819 us; error 64 us; skew 500 ppm
I20260808 15:48:55.150177 3412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:55.152490 3412 webserver.cc:460] Webserver started at http://127.27.84.195:43707/ using document root <none> and password file <none>
I20260808 15:48:55.153187 3412 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:55.153276 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:55.153577 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:55.156713 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "81653c0dec0b4db2b92e93a74e692e78"
format_stamp: "Formatted at 2026-08-08 15:48:55 on dist-test-slave-twwt"
server_key: "60f124073297425661f6350f00ce2aef"
server_key_iv: "8e58325193686b53f389b19b1b924d24"
server_key_version: "encryptionkey@0"
I20260808 15:48:55.157513 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "81653c0dec0b4db2b92e93a74e692e78"
format_stamp: "Formatted at 2026-08-08 15:48:55 on dist-test-slave-twwt"
server_key: "60f124073297425661f6350f00ce2aef"
server_key_iv: "8e58325193686b53f389b19b1b924d24"
server_key_version: "encryptionkey@0"
I20260808 15:48:55.162968 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:48:55.166780 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:55.168251 3412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:55.168454 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "81653c0dec0b4db2b92e93a74e692e78"
format_stamp: "Formatted at 2026-08-08 15:48:55 on dist-test-slave-twwt"
server_key: "60f124073297425661f6350f00ce2aef"
server_key_iv: "8e58325193686b53f389b19b1b924d24"
server_key_version: "encryptionkey@0"
I20260808 15:48:55.168695 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:55.174728 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:55.175575 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:55.175866 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:55.176872 3412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:48:55.178351 3412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:48:55.178483 3412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:55.178614 3412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:48:55.178727 3412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 15:48:55.213130 3412 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:38943
I20260808 15:48:55.213191 3608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:38943 every 8 connection(s)
I20260808 15:48:55.214557 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:48:55.215624 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 3412
I20260808 15:48:55.215802 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:48:55.216434 27987 external_mini_cluster.cc:1428] Setting key 4adb0e2d18bd687c4bdc1f252ae400c5
I20260808 15:48:55.223673 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.217704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:52643 (local address 127.27.84.254:43623)
0808 15:48:55.217932 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.217938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.219494 (+ 1556us) server_negotiation.cc:407] Connection header received
0808 15:48:55.221416 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.221422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.221492 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.221606 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.222916 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.222922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.222927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.222946 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.222986 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.222997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.223117 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.223310 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.223383 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:48:55.224670 3613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.218336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43623 (local address 127.27.84.195:52643)
0808 15:48:55.219337 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.219374 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:55.221148 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:55.221845 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:55.221853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:55.221947 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:55.222572 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:55.222574 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:55.222668 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:55.222759 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:55.223092 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:55.223104 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:55.223111 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:48:55.223410 (+ 299us) client_negotiation.cc:240] Negotiation successful
0808 15:48:55.223676 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":346,"threads_started":1}
I20260808 15:48:55.225955 3610 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43623
I20260808 15:48:55.226250 3610 heartbeater.cc:464] Registering TS with master...
I20260808 15:48:55.226899 3610 heartbeater.cc:511] Master 127.27.84.254:43623 requested a full tablet report, sending...
I20260808 15:48:55.228227 2875 ts_manager.cc:194] Registered new tserver with Master: 81653c0dec0b4db2b92e93a74e692e78 (127.27.84.195:38943)
I20260808 15:48:55.232091 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:48:55.239267 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.236711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.27.84.254:43623)
0808 15:48:55.236902 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.236911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.237320 (+ 409us) server_negotiation.cc:407] Connection header received
0808 15:48:55.237522 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.237528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.237591 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.237692 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.238572 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.238577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.238582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.238601 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.238662 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.238677 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.238828 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.238920 (+ 92us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.238986 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:48:55.243129 2875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:48:55.250154 2875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:48:55.281731 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.262889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49782 (local address 127.27.84.194:46153)
0808 15:48:55.275555 (+ 12666us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.275568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.276415 (+ 847us) server_negotiation.cc:407] Connection header received
0808 15:48:55.276745 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.276755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.277026 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.277187 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.280626 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.280636 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.280652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.280675 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.280744 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.280768 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.280888 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.281236 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.281369 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12443,"thread_start_us":12680,"threads_started":1}
I20260808 15:48:55.282554 3630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.262772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:46153 (local address 127.0.0.1:49782)
0808 15:48:55.276226 (+ 13454us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.276269 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:55.276440 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:55.278331 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:55.278340 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:55.278445 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:55.280286 (+ 1841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:55.280290 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:55.280331 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:55.280417 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:55.280999 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:55.281012 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:55.281020 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:48:55.282005 (+ 985us) client_negotiation.cc:240] Negotiation successful
0808 15:48:55.282061 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13165,"thread_start_us":13199,"threads_started":1}
I20260808 15:48:55.283010 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.263081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.27.84.195:38943)
0808 15:48:55.276717 (+ 13636us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.276725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.276758 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:48:55.276821 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.276827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.277025 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.277187 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.281273 (+ 4086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.281283 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.281295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.281314 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.281383 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.281406 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.281585 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.282201 (+ 616us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.282303 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13424,"thread_start_us":13455,"threads_started":1}
I20260808 15:48:55.283303 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.262864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:38943 (local address 127.0.0.1:45508)
0808 15:48:55.276300 (+ 13436us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.276330 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:55.276559 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:55.278331 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:55.278340 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:55.278445 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:55.281026 (+ 2581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:55.281029 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:55.281046 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:55.281092 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:55.281651 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:55.281658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:55.281665 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:48:55.281798 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 15:48:55.281912 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13261,"thread_start_us":12868,"threads_started":1}
I20260808 15:48:55.285131 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.277141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60172)
0808 15:48:55.280255 (+ 3114us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.280274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:55.280391 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:55.282213 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:55.282217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:55.282239 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:55.283711 (+ 1472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:55.283716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:55.283734 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:55.283784 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:55.284476 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:55.284483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:55.284491 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:48:55.284624 (+ 133us) client_negotiation.cc:240] Negotiation successful
0808 15:48:55.284678 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2964,"thread_start_us":2303,"threads_started":1}
I20260808 15:48:55.285826 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.277180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.27.84.193:42415)
0808 15:48:55.279101 (+ 1921us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.279107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.280524 (+ 1417us) server_negotiation.cc:407] Connection header received
0808 15:48:55.280602 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.280610 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.280864 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.281024 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.283974 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.283985 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.284000 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.284107 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.284181 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.284208 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.284409 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.285354 (+ 945us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.285443 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1789,"thread_start_us":1671,"threads_started":1}
I20260808 15:48:55.287665 3290 tablet_service.cc:1467] Processing CreateTablet for tablet 0260389112ed4a3b85bd8cb3831c785a (DEFAULT_TABLE table=test-table [id=2126f9fb49104e919b8e43438326949e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:55.288352 3505 tablet_service.cc:1467] Processing CreateTablet for tablet 0260389112ed4a3b85bd8cb3831c785a (DEFAULT_TABLE table=test-table [id=2126f9fb49104e919b8e43438326949e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:55.289322 3290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0260389112ed4a3b85bd8cb3831c785a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:55.289758 3505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0260389112ed4a3b85bd8cb3831c785a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:55.289978 3082 tablet_service.cc:1467] Processing CreateTablet for tablet 0260389112ed4a3b85bd8cb3831c785a (DEFAULT_TABLE table=test-table [id=2126f9fb49104e919b8e43438326949e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:48:55.291111 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0260389112ed4a3b85bd8cb3831c785a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:55.300078 3189 heartbeater.cc:503] Master 127.27.84.254:43623 was elected leader, sending a full tablet report...
I20260808 15:48:55.302827 3637 tablet_bootstrap.cc:492] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: Bootstrap starting.
I20260808 15:48:55.307055 3637 tablet_bootstrap.cc:654] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:55.307403 3641 tablet_bootstrap.cc:492] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Bootstrap starting.
I20260808 15:48:55.308646 3637 log.cc:824] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:55.310182 3640 tablet_bootstrap.cc:492] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Bootstrap starting.
I20260808 15:48:55.311589 3641 tablet_bootstrap.cc:654] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:55.311965 3637 tablet_bootstrap.cc:492] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: No bootstrap required, opened a new log
I20260808 15:48:55.312276 3637 ts_tablet_manager.cc:1397] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 15:48:55.313006 3641 log.cc:824] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:55.313980 3640 tablet_bootstrap.cc:654] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:55.315249 3640 log.cc:824] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:55.316195 3641 tablet_bootstrap.cc:492] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: No bootstrap required, opened a new log
I20260808 15:48:55.316488 3641 ts_tablet_manager.cc:1397] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:48:55.318423 3640 tablet_bootstrap.cc:492] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: No bootstrap required, opened a new log
I20260808 15:48:55.318728 3640 ts_tablet_manager.cc:1397] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:48:55.322954 3637 raft_consensus.cc:348] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.323469 3637 raft_consensus.cc:374] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:55.323581 3637 raft_consensus.cc:729] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269a26ae1d79435384f3335d462e4c04, State: Initialized, Role: FOLLOWER
I20260808 15:48:55.324333 3637 consensus_queue.cc:260] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.326565 3640 raft_consensus.cc:348] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.327121 3640 raft_consensus.cc:374] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:55.327237 3640 raft_consensus.cc:729] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1236070f1c154cbc9726a06528e51197, State: Initialized, Role: FOLLOWER
I20260808 15:48:55.327798 3641 raft_consensus.cc:348] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.327935 3640 consensus_queue.cc:260] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.328189 3637 ts_tablet_manager.cc:1428] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260808 15:48:55.328279 3641 raft_consensus.cc:374] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:55.328382 3641 raft_consensus.cc:729] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81653c0dec0b4db2b92e93a74e692e78, State: Initialized, Role: FOLLOWER
I20260808 15:48:55.329072 3641 consensus_queue.cc:260] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.330435 3408 heartbeater.cc:503] Master 127.27.84.254:43623 was elected leader, sending a full tablet report...
I20260808 15:48:55.331243 3640 ts_tablet_manager.cc:1428] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260808 15:48:55.332374 3610 heartbeater.cc:503] Master 127.27.84.254:43623 was elected leader, sending a full tablet report...
I20260808 15:48:55.334978 3641 ts_tablet_manager.cc:1428] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Time spent starting tablet: real 0.018s user 0.012s sys 0.005s
I20260808 15:48:55.427722 3647 raft_consensus.cc:482] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:48:55.428058 3647 raft_consensus.cc:504] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.430019 3647 leader_election.cc:288] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81653c0dec0b4db2b92e93a74e692e78 (127.27.84.195:38943), 269a26ae1d79435384f3335d462e4c04 (127.27.84.193:42415)
I20260808 15:48:55.433519 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.430478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:54499 (local address 127.27.84.195:38943)
0808 15:48:55.430710 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.430715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.431204 (+ 489us) server_negotiation.cc:407] Connection header received
0808 15:48:55.431527 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.431533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.431595 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.431676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.432868 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.432874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.432881 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.432905 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.432943 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.432960 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.433096 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.433210 (+ 114us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.433258 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:48:55.433921 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.431430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:51733 (local address 127.27.84.193:42415)
0808 15:48:55.431631 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.431638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.431888 (+ 250us) server_negotiation.cc:407] Connection header received
0808 15:48:55.432082 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.432088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.432145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.432244 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.433190 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.433199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.433206 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.433230 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.433280 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.433296 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.433423 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.433608 (+ 185us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.433675 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:48:55.434211 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.430275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:38943 (local address 127.27.84.194:54499)
0808 15:48:55.431094 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.431117 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:55.431318 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:55.431858 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:55.431861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:55.431881 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:55.432627 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:55.432632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:55.432653 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:55.432714 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:55.433061 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:55.433065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:55.433068 (+ 3us) client_negotiation.cc:769] Sending connection context
0808 15:48:55.433253 (+ 185us) client_negotiation.cc:240] Negotiation successful
0808 15:48:55.433309 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":378,"threads_started":1}
I20260808 15:48:55.434244 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.431156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.27.84.194:51733)
0808 15:48:55.431775 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.431789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:55.431942 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:55.432275 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:55.432282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:55.432304 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:48:55.432965 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:48:55.432968 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:48:55.432985 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:48:55.433033 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:48:55.433408 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:48:55.433412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:48:55.433416 (+ 4us) client_negotiation.cc:769] Sending connection context
0808 15:48:55.433708 (+ 292us) client_negotiation.cc:240] Negotiation successful
0808 15:48:55.433788 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":305,"threads_started":1}
I20260808 15:48:55.435282 3545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0260389112ed4a3b85bd8cb3831c785a" candidate_uuid: "1236070f1c154cbc9726a06528e51197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81653c0dec0b4db2b92e93a74e692e78" is_pre_election: true
I20260808 15:48:55.435292 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0260389112ed4a3b85bd8cb3831c785a" candidate_uuid: "1236070f1c154cbc9726a06528e51197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269a26ae1d79435384f3335d462e4c04" is_pre_election: true
I20260808 15:48:55.435782 3545 raft_consensus.cc:2454] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1236070f1c154cbc9726a06528e51197 in term 0.
I20260808 15:48:55.435798 3115 raft_consensus.cc:2454] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1236070f1c154cbc9726a06528e51197 in term 0.
I20260808 15:48:55.436401 3216 leader_election.cc:302] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1236070f1c154cbc9726a06528e51197, 269a26ae1d79435384f3335d462e4c04; no voters:
I20260808 15:48:55.436776 3647 raft_consensus.cc:2781] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:48:55.436928 3647 raft_consensus.cc:482] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:48:55.437042 3647 raft_consensus.cc:3037] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:55.439415 3647 raft_consensus.cc:504] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.440119 3647 leader_election.cc:288] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [CANDIDATE]: Term 1 election: Requested vote from peers 81653c0dec0b4db2b92e93a74e692e78 (127.27.84.195:38943), 269a26ae1d79435384f3335d462e4c04 (127.27.84.193:42415)
I20260808 15:48:55.440490 3545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0260389112ed4a3b85bd8cb3831c785a" candidate_uuid: "1236070f1c154cbc9726a06528e51197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81653c0dec0b4db2b92e93a74e692e78"
I20260808 15:48:55.440618 3115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0260389112ed4a3b85bd8cb3831c785a" candidate_uuid: "1236070f1c154cbc9726a06528e51197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269a26ae1d79435384f3335d462e4c04"
I20260808 15:48:55.440704 3545 raft_consensus.cc:3037] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:55.440834 3115 raft_consensus.cc:3037] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:55.444126 3115 raft_consensus.cc:2454] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1236070f1c154cbc9726a06528e51197 in term 1.
I20260808 15:48:55.444126 3545 raft_consensus.cc:2454] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1236070f1c154cbc9726a06528e51197 in term 1.
I20260808 15:48:55.444504 3216 leader_election.cc:302] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1236070f1c154cbc9726a06528e51197, 81653c0dec0b4db2b92e93a74e692e78; no voters:
I20260808 15:48:55.444796 3647 raft_consensus.cc:2781] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:55.445217 3647 raft_consensus.cc:686] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 1 LEADER]: Becoming Leader. State: Replica: 1236070f1c154cbc9726a06528e51197, State: Running, Role: LEADER
I20260808 15:48:55.445739 3647 consensus_queue.cc:237] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } }
I20260808 15:48:55.450443 2874 catalog_manager.cc:5495] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1236070f1c154cbc9726a06528e51197 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "1236070f1c154cbc9726a06528e51197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1236070f1c154cbc9726a06528e51197" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 46153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 } health_report { overall_health: UNKNOWN } } }
I20260808 15:48:55.465998 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.463945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.27.84.194:46153)
0808 15:48:55.464130 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.464135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.464153 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:48:55.464292 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.464296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.464384 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.464486 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.465424 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.465428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.465432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.465448 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.465481 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.465494 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.465585 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.465693 (+ 108us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.465752 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260808 15:48:55.466533 3612 tablet.cc:2369] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:48:55.475253 3545 raft_consensus.cc:1261] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 1 FOLLOWER]: Refusing update from remote peer 1236070f1c154cbc9726a06528e51197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:55.475250 3115 raft_consensus.cc:1261] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 1 FOLLOWER]: Refusing update from remote peer 1236070f1c154cbc9726a06528e51197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:48:55.476305 3654 consensus_queue.cc:1035] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269a26ae1d79435384f3335d462e4c04" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:55.476711 3647 consensus_queue.cc:1035] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81653c0dec0b4db2b92e93a74e692e78" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:48:55.488864 3658 mvcc.cc:205] Tried to move back new op lower bound from 7316292138895048704 to 7316292138789957632. Current Snapshot: MvccSnapshot[applied={T|T < 7316292138895048704}]
I20260808 15:48:55.489818 3659 mvcc.cc:205] Tried to move back new op lower bound from 7316292138895048704 to 7316292138789957632. Current Snapshot: MvccSnapshot[applied={T|T < 7316292138895048704}]
I20260808 15:48:55.503206 2871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:55.503561 2871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:48:55.507795 2871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 529a04ce6eab4aa1983a3d536c7b14b7: Sending DeleteTablet for 3 replicas of tablet 0260389112ed4a3b85bd8cb3831c785a
I20260808 15:48:55.508901 3290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0260389112ed4a3b85bd8cb3831c785a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:55 UTC) from {username='slave'} at 127.0.0.1:49782
I20260808 15:48:55.509042 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet 0260389112ed4a3b85bd8cb3831c785a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:55 UTC) from {username='slave'} at 127.0.0.1:60172
I20260808 15:48:55.509123 3505 tablet_service.cc:1514] Processing DeleteTablet for tablet 0260389112ed4a3b85bd8cb3831c785a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:48:55 UTC) from {username='slave'} at 127.0.0.1:45508
I20260808 15:48:55.509976 3679 tablet_replica.cc:330] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: stopping tablet replica
I20260808 15:48:55.510560 3679 raft_consensus.cc:2229] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:48:55.510725 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 2976
I20260808 15:48:55.511888 3679 raft_consensus.cc:2258] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:55.514562 3679 ts_tablet_manager.cc:1905] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:55.524415 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.512293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.27.84.193:42415)
0808 15:48:55.512487 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.512494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.521719 (+ 9225us) server_negotiation.cc:407] Connection header received
0808 15:48:55.521937 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.521947 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.522093 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.522189 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.523532 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.523539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.523544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.523565 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.523604 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.523620 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.523729 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.524012 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.524091 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:48:55.525271 3679 ts_tablet_manager.cc:1918] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:55.525489 3679 log.cc:1196] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/0260389112ed4a3b85bd8cb3831c785a
I20260808 15:48:55.525705 3183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:55.526211 3679 ts_tablet_manager.cc:1939] T 0260389112ed4a3b85bd8cb3831c785a P 1236070f1c154cbc9726a06528e51197: Deleting consensus metadata
I20260808 15:48:55.528592 2852 catalog_manager.cc:4841] TS 1236070f1c154cbc9726a06528e51197 (127.27.84.194:46153): tablet 0260389112ed4a3b85bd8cb3831c785a (table test-table [id=2126f9fb49104e919b8e43438326949e]) successfully deleted
I20260808 15:48:55.532989 3683 tablet_replica.cc:330] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: stopping tablet replica
I20260808 15:48:55.533484 3683 raft_consensus.cc:2229] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:55.534008 3683 raft_consensus.cc:2258] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:55.536540 3683 ts_tablet_manager.cc:1905] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:48:55.544731 3683 ts_tablet_manager.cc:1918] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:48:55.544929 3683 log.cc:1196] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/0260389112ed4a3b85bd8cb3831c785a
I20260808 15:48:55.545650 3683 ts_tablet_manager.cc:1939] T 0260389112ed4a3b85bd8cb3831c785a P 81653c0dec0b4db2b92e93a74e692e78: Deleting consensus metadata
I20260808 15:48:55.547569 2850 catalog_manager.cc:4841] TS 81653c0dec0b4db2b92e93a74e692e78 (127.27.84.195:38943): tablet 0260389112ed4a3b85bd8cb3831c785a (table test-table [id=2126f9fb49104e919b8e43438326949e]) successfully deleted
I20260808 15:48:55.629398 3682 tablet_replica.cc:330] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04: stopping tablet replica
I20260808 15:48:55.629844 3682 raft_consensus.cc:2229] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:48:55.630342 3682 raft_consensus.cc:2258] T 0260389112ed4a3b85bd8cb3831c785a P 269a26ae1d79435384f3335d462e4c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:48:55.631120 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 2976
I20260808 15:48:55.657897 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 3193
W20260808 15:48:55.657802 2850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:42415 (error 108)
I20260808 15:48:55.660080 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.659503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60184)
0808 15:48:55.659728 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.659890 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260808 15:48:55.660434 2850 catalog_manager.cc:4584] TS 269a26ae1d79435384f3335d462e4c04 (127.27.84.193:42415): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0260389112ed4a3b85bd8cb3831c785a: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
I20260808 15:48:55.661754 3631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.658877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.27.84.194:46153)
0808 15:48:55.659091 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.659097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.659716 (+ 619us) server_negotiation.cc:407] Connection header received
0808 15:48:55.659826 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.659830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.659889 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.660000 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.661068 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.661073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.661076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.661093 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.661126 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.661137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.661270 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.661424 (+ 154us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.661491 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:48:55.662536 3401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:55.714560 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.713958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60186)
0808 15:48:55.714211 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.714355 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 15:48:55.715711 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.715314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60194)
0808 15:48:55.715482 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.715568 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260808 15:48:55.776413 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.775792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60198)
0808 15:48:55.776015 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.776183 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:48:55.778033 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 3193
I20260808 15:48:55.778218 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.777816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60214)
0808 15:48:55.777975 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.778059 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260808 15:48:55.803946 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 3412
I20260808 15:48:55.807158 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.804748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.27.84.195:38943)
0808 15:48:55.804947 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:55.804952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:55.805014 (+ 62us) server_negotiation.cc:407] Connection header received
0808 15:48:55.805173 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:55.805177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:55.805253 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:55.805520 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:55.806394 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:55.806398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:55.806401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:55.806416 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:55.806448 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:55.806459 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:55.806569 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:55.806786 (+ 217us) server_negotiation.cc:299] Negotiation successful
0808 15:48:55.806882 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:48:55.807837 3602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:55.858886 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.858330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60220)
0808 15:48:55.858571 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.858718 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 15:48:55.860092 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:55.859660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60222)
0808 15:48:55.859885 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:55.859956 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 15:48:55.898517 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 3412
I20260808 15:48:55.923110 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 2821
I20260808 15:48:55.924400 2921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:56.041061 3633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:56.039365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42415 (local address 127.0.0.1:60224)
0808 15:48:56.040240 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:56.040387 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:42415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":679}
I20260808 15:48:56.041060 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 2821
2026-08-08T15:48:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2849 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260808 15:48:56.092542 27987 mini_oidc.cc:152] Starting JWKS server
I20260808 15:48:56.095433 27987 webserver.cc:460] Webserver started at https://127.0.0.1:44509/ using document root <none> and password file <none>
I20260808 15:48:56.095777 27987 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:48:56.096464 27987 webserver.cc:460] Webserver started at http://127.0.0.1:45085/ using document root <none> and password file <none>
2026-08-08T15:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:56Z Disabled control of system clock
I20260808 15:48:56.135450 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43075
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:36483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43075
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44509/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/testchainca.cert with env {}
W20260808 15:48:56.484617 3697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:56.484968 3697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:56.485131 3697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:56.492177 3697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:56.492322 3697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:56.492413 3697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:56.492504 3697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:56.492573 3697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:48:56.492661 3697 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:48:56.492758 3697 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44509/jwks.json
I20260808 15:48:56.502663 3697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:36483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44509/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:56.504027 3697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:56.505340 3697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:56.517396 3704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:56.517305 3710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:56.517325 3706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:56.518702 3697 server_base.cc:1034] running on GCE node
I20260808 15:48:56.519511 3697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:56.520987 3697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:56.522202 3697 hybrid_clock.cc:648] HybridClock initialized: now 1786204136522139 us; error 52 us; skew 500 ppm
I20260808 15:48:56.522557 3697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:56.528497 3697 webserver.cc:460] Webserver started at http://127.27.84.254:38209/ using document root <none> and password file <none>
I20260808 15:48:56.529232 3697 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:56.529364 3697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:56.529695 3697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:56.532958 3697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "5bd66e85b2b341c28c13af1daba15f9f"
format_stamp: "Formatted at 2026-08-08 15:48:56 on dist-test-slave-twwt"
server_key: "9d7ff24325eebf7135e52cec85a37fab"
server_key_iv: "f4f9cdbc1439dc802cd947fbed0bd1ec"
server_key_version: "encryptionkey@0"
I20260808 15:48:56.533812 3697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "5bd66e85b2b341c28c13af1daba15f9f"
format_stamp: "Formatted at 2026-08-08 15:48:56 on dist-test-slave-twwt"
server_key: "9d7ff24325eebf7135e52cec85a37fab"
server_key_iv: "f4f9cdbc1439dc802cd947fbed0bd1ec"
server_key_version: "encryptionkey@0"
I20260808 15:48:56.539147 3697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260808 15:48:56.542707 3720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:56.544076 3697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:48:56.544284 3697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "5bd66e85b2b341c28c13af1daba15f9f"
format_stamp: "Formatted at 2026-08-08 15:48:56 on dist-test-slave-twwt"
server_key: "9d7ff24325eebf7135e52cec85a37fab"
server_key_iv: "f4f9cdbc1439dc802cd947fbed0bd1ec"
server_key_version: "encryptionkey@0"
I20260808 15:48:56.544512 3697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:56.559808 3697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:56.560655 3697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:56.560990 3697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:56.580296 3697 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43075
I20260808 15:48:56.580317 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43075 every 8 connection(s)
I20260808 15:48:56.582005 3697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:56.586176 3792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:56.590431 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 3697
I20260808 15:48:56.590693 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:56.591536 27987 external_mini_cluster.cc:1428] Setting key b755d8690fc4955b1fcf06c6af895581
I20260808 15:48:56.600998 3792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f: Bootstrap starting.
I20260808 15:48:56.605010 3792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:56.606129 3792 log.cc:824] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:56.606187 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:56.593656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.27.84.254:43075)
0808 15:48:56.594808 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:56.594821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:56.594869 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:48:56.596567 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:56.596595 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:56.597142 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:56.597533 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:56.598202 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:56.599620 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:56.601519 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:56.602228 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:56.602379 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:56.603793 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:56.603800 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:56.603813 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:56.604375 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:56.604471 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:56.604492 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:56.604579 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:56.604824 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 15:48:56.605201 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":402,"threads_started":1}
I20260808 15:48:56.609301 3792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f: No bootstrap required, opened a new log
I20260808 15:48:56.610183 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:48:56.617008 3792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd66e85b2b341c28c13af1daba15f9f" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43075 } }
I20260808 15:48:56.617394 3792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:56.617512 3792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd66e85b2b341c28c13af1daba15f9f, State: Initialized, Role: FOLLOWER
I20260808 15:48:56.618083 3792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd66e85b2b341c28c13af1daba15f9f" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43075 } }
I20260808 15:48:56.618311 3792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:56.618448 3792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:56.618645 3792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:56.621083 3792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd66e85b2b341c28c13af1daba15f9f" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43075 } }
I20260808 15:48:56.621637 3792 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd66e85b2b341c28c13af1daba15f9f; no voters:
I20260808 15:48:56.622228 3792 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:56.622377 3800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:56.623157 3800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [term 1 LEADER]: Becoming Leader. State: Replica: 5bd66e85b2b341c28c13af1daba15f9f, State: Running, Role: LEADER
I20260808 15:48:56.623682 3800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd66e85b2b341c28c13af1daba15f9f" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43075 } }
I20260808 15:48:56.624379 3792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:56.626452 3802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd66e85b2b341c28c13af1daba15f9f. Latest consensus state: current_term: 1 leader_uuid: "5bd66e85b2b341c28c13af1daba15f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd66e85b2b341c28c13af1daba15f9f" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43075 } } }
I20260808 15:48:56.626894 3802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:56.626828 3801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd66e85b2b341c28c13af1daba15f9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd66e85b2b341c28c13af1daba15f9f" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43075 } } }
I20260808 15:48:56.627125 3801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:56.630735 3806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:56.636699 3806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:56.645794 3806 catalog_manager.cc:1347] Generated new cluster ID: 3e7032206206493cb51cc6e79f3aeaec
I20260808 15:48:56.645962 3806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:56.687542 3806 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:56.688948 3806 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:56.705688 3806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd66e85b2b341c28c13af1daba15f9f: Generated new TSK 0
I20260808 15:48:56.706588 3806 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 15:48:56.746140 3846 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260808 15:48:56.746380 3846 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:48:56.749011 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:56.724272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.27.84.254:43075)
0808 15:48:56.724546 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:56.724556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:56.725200 (+ 644us) server_negotiation.cc:407] Connection header received
0808 15:48:56.725560 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:56.725566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:56.725664 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:56.725861 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:48:56.726513 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:56.727750 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:56.729208 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:56.729943 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:56.730075 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:56.730603 (+ 528us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:48:56.748200 (+ 17597us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 15:48:56.748688 (+ 488us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260808 15:48:56.749150 3796 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48410: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260808 15:48:56.750707 3845 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:56.724163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43075 (local address 127.0.0.1:48410)
0808 15:48:56.725093 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:56.725111 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:56.725272 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:56.725953 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:56.725957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:56.725971 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:48:56.726360 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:56.726370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:56.727922 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:56.727926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:56.729060 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:56.729072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:56.730120 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:56.730124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:56.730255 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:56.730313 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:48:56.748606 (+ 18293us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0808 15:48:56.748938 (+ 332us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.84.254:43075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":587,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.84.254:43075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44509/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260808 15:48:56.752403 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 3697
I20260808 15:48:56.753716 3785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:56.855082 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 3697
2026-08-08T15:48:56Z chronyd exiting
I20260808 15:48:56.916117 27987 test_util.cc:175] -----------------------------------------------
I20260808 15:48:56.916320 27987 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786204089698720-27987-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (828 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260808 15:48:56.920377 27987 mini_oidc.cc:152] Starting JWKS server
I20260808 15:48:56.922286 27987 webserver.cc:460] Webserver started at https://127.0.0.1:42265/ using document root <none> and password file <none>
I20260808 15:48:56.922520 27987 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:48:56.923234 27987 webserver.cc:460] Webserver started at http://127.0.0.1:39779/ using document root <none> and password file <none>
2026-08-08T15:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:56Z Disabled control of system clock
I20260808 15:48:56.960800 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:46097
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:46097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/testchainca.cert with env {}
W20260808 15:48:57.301743 3855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:57.302089 3855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:57.302256 3855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:57.309541 3855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:57.309677 3855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:57.309751 3855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:57.309855 3855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:57.309927 3855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:48:57.310015 3855 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:48:57.310066 3855 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42265/jwks.json
I20260808 15:48:57.320327 3855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:46097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:57.321995 3855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:57.323546 3855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:57.335672 3860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:57.335713 3864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:57.336207 3861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:57.336861 3855 server_base.cc:1034] running on GCE node
I20260808 15:48:57.337658 3855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:57.339147 3855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:57.340372 3855 hybrid_clock.cc:648] HybridClock initialized: now 1786204137340318 us; error 36 us; skew 500 ppm
I20260808 15:48:57.340760 3855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:57.343139 3855 webserver.cc:460] Webserver started at http://127.27.84.254:35445/ using document root <none> and password file <none>
I20260808 15:48:57.343854 3855 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:57.343978 3855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:57.344314 3855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:57.347348 3855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "6a0bb47cfa114a2f819b806b4b53d16d"
format_stamp: "Formatted at 2026-08-08 15:48:57 on dist-test-slave-twwt"
server_key: "6eb90d1a3970675a8d4d56ae01cf1643"
server_key_iv: "6c971d79a83028a027200fc41e339120"
server_key_version: "encryptionkey@0"
I20260808 15:48:57.348181 3855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "6a0bb47cfa114a2f819b806b4b53d16d"
format_stamp: "Formatted at 2026-08-08 15:48:57 on dist-test-slave-twwt"
server_key: "6eb90d1a3970675a8d4d56ae01cf1643"
server_key_iv: "6c971d79a83028a027200fc41e339120"
server_key_version: "encryptionkey@0"
I20260808 15:48:57.353480 3855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:48:57.357097 3872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:57.358554 3855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:48:57.358775 3855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "6a0bb47cfa114a2f819b806b4b53d16d"
format_stamp: "Formatted at 2026-08-08 15:48:57 on dist-test-slave-twwt"
server_key: "6eb90d1a3970675a8d4d56ae01cf1643"
server_key_iv: "6c971d79a83028a027200fc41e339120"
server_key_version: "encryptionkey@0"
I20260808 15:48:57.359025 3855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:57.375663 3855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:57.376550 3855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:57.377064 3855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:57.396394 3855 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:46097
I20260808 15:48:57.396423 3959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:46097 every 8 connection(s)
I20260808 15:48:57.397877 3855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:57.401860 3961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:57.404130 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 3855
I20260808 15:48:57.404331 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:57.405143 27987 external_mini_cluster.cc:1428] Setting key 44932730135a4d70a7677c842be53c69
I20260808 15:48:57.414867 3961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d: Bootstrap starting.
I20260808 15:48:57.418829 3961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:57.419867 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:57.407202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.27.84.254:46097)
0808 15:48:57.408340 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:57.408352 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:57.408396 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:48:57.410037 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:57.410059 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:57.410540 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:57.410941 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:57.411595 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:57.413159 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:57.415096 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:57.415817 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:57.415957 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:57.417270 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:57.417277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:57.417291 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:57.417937 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:57.418036 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:57.418059 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:57.418149 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:57.418404 (+ 255us) server_negotiation.cc:299] Negotiation successful
0808 15:48:57.418792 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":436,"threads_started":1}
I20260808 15:48:57.419993 3961 log.cc:824] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:57.423593 3961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d: No bootstrap required, opened a new log
I20260808 15:48:57.424340 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:48:57.431917 3961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 46097 } }
I20260808 15:48:57.432360 3961 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:57.432483 3961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a0bb47cfa114a2f819b806b4b53d16d, State: Initialized, Role: FOLLOWER
I20260808 15:48:57.433131 3961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 46097 } }
I20260808 15:48:57.433389 3961 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:57.433538 3961 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:57.433712 3961 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:57.436419 3961 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 46097 } }
I20260808 15:48:57.436932 3961 leader_election.cc:302] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a0bb47cfa114a2f819b806b4b53d16d; no voters:
I20260808 15:48:57.437543 3961 leader_election.cc:288] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:57.437793 3968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:57.438858 3968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [term 1 LEADER]: Becoming Leader. State: Replica: 6a0bb47cfa114a2f819b806b4b53d16d, State: Running, Role: LEADER
I20260808 15:48:57.439365 3968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 46097 } }
I20260808 15:48:57.439658 3961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:57.442158 3972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a0bb47cfa114a2f819b806b4b53d16d. Latest consensus state: current_term: 1 leader_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 46097 } } }
I20260808 15:48:57.442312 3971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a0bb47cfa114a2f819b806b4b53d16d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 46097 } } }
I20260808 15:48:57.442557 3972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:57.442644 3971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:57.445585 3982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:57.450347 3982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:57.458957 3982 catalog_manager.cc:1347] Generated new cluster ID: 27ade50806f9495bb8cb5cd2d9feda36
I20260808 15:48:57.459085 3982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:57.467913 3982 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:57.469374 3982 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:57.492007 3982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a0bb47cfa114a2f819b806b4b53d16d: Generated new TSK 0
I20260808 15:48:57.493160 3982 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260808 15:48:57.543062 4008 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260808 15:48:57.543211 4008 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:48:57.544546 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:57.517756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36052 (local address 127.27.84.254:46097)
0808 15:48:57.517948 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:57.517954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:57.518644 (+ 690us) server_negotiation.cc:407] Connection header received
0808 15:48:57.518882 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:57.518887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:57.518968 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:57.519099 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:48:57.519648 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:57.520376 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:57.522389 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:57.522915 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:57.523039 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:57.523630 (+ 591us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:48:57.543814 (+ 20184us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 15:48:57.544238 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260808 15:48:57.544638 4006 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:57.517737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:46097 (local address 127.0.0.1:36052)
0808 15:48:57.518490 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:57.518512 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:57.518699 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:57.519145 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:57.519150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:57.519164 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:48:57.519476 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:57.519488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:57.520523 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:57.520529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:57.522240 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:57.522257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:57.523118 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:57.523125 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:57.523324 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:57.523371 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:48:57.544085 (+ 20714us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0808 15:48:57.544321 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.84.254:46097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":438,"threads_started":1}
W20260808 15:48:57.544692 3965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36052: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260808 15:48:57.546110 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 3855
I20260808 15:48:57.547415 3952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:57.642939 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 3855
2026-08-08T15:48:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (786 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260808 15:48:57.706884 27987 mini_oidc.cc:152] Starting JWKS server
I20260808 15:48:57.709179 27987 webserver.cc:460] Webserver started at https://127.0.0.1:42471/ using document root <none> and password file <none>
I20260808 15:48:57.709581 27987 mini_oidc.cc:189] Starting OIDC Discovery server
I20260808 15:48:57.710253 27987 webserver.cc:460] Webserver started at http://127.0.0.1:35167/ using document root <none> and password file <none>
2026-08-08T15:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:57Z Disabled control of system clock
I20260808 15:48:57.752455 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:40955
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:37723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:40955
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42471/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 15:48:58.105633 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:58.105943 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:58.106031 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:58.113409 4022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:58.113528 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:58.113566 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:58.113626 4022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:58.113670 4022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:48:58.113732 4022 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:48:58.113773 4022 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42471/jwks.json
I20260808 15:48:58.123332 4022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:40955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:40955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42471/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:58.125008 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:58.127084 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:58.139441 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:58.139546 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:58.139772 4028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:58.141045 4022 server_base.cc:1034] running on GCE node
I20260808 15:48:58.141870 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:58.143544 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:58.144749 4022 hybrid_clock.cc:648] HybridClock initialized: now 1786204138144706 us; error 35 us; skew 500 ppm
I20260808 15:48:58.145079 4022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:58.147444 4022 webserver.cc:460] Webserver started at http://127.27.84.254:39193/ using document root <none> and password file <none>
I20260808 15:48:58.148114 4022 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:58.148202 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:58.148497 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:58.151634 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "60355e4477ed4484960c7befa8fc2433"
format_stamp: "Formatted at 2026-08-08 15:48:58 on dist-test-slave-twwt"
server_key: "f5976852ab306447e6513bd3865999ac"
server_key_iv: "647666212e84114667563c9217eedab6"
server_key_version: "encryptionkey@0"
I20260808 15:48:58.152565 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "60355e4477ed4484960c7befa8fc2433"
format_stamp: "Formatted at 2026-08-08 15:48:58 on dist-test-slave-twwt"
server_key: "f5976852ab306447e6513bd3865999ac"
server_key_iv: "647666212e84114667563c9217eedab6"
server_key_version: "encryptionkey@0"
I20260808 15:48:58.158046 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:48:58.161809 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:58.163630 4022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:48:58.163820 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "60355e4477ed4484960c7befa8fc2433"
format_stamp: "Formatted at 2026-08-08 15:48:58 on dist-test-slave-twwt"
server_key: "f5976852ab306447e6513bd3865999ac"
server_key_iv: "647666212e84114667563c9217eedab6"
server_key_version: "encryptionkey@0"
I20260808 15:48:58.164111 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:58.190306 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:58.191236 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:58.191581 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:58.210933 4022 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:40955
I20260808 15:48:58.210976 4122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:40955 every 8 connection(s)
I20260808 15:48:58.212354 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:58.216249 4123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:58.221064 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 4022
I20260808 15:48:58.221297 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:58.222131 27987 external_mini_cluster.cc:1428] Setting key dfbd4278811a4e6dcc7b11f9ac73b386
I20260808 15:48:58.228385 4123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433: Bootstrap starting.
I20260808 15:48:58.231588 4123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:58.232674 4123 log.cc:824] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:58.236274 4123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433: No bootstrap required, opened a new log
I20260808 15:48:58.236495 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:58.224159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.27.84.254:40955)
0808 15:48:58.225204 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:58.225215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:58.225260 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:48:58.227116 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:58.227139 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:58.227592 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:58.227980 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:58.228573 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.229897 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.231308 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.232113 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.232326 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:58.233596 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:58.233603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:58.233616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:58.234274 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:58.234373 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:58.234404 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:58.234534 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:58.234846 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 15:48:58.235181 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":410,"threads_started":1}
I20260808 15:48:58.240708 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:48:58.246443 4123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60355e4477ed4484960c7befa8fc2433" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40955 } }
I20260808 15:48:58.246906 4123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:58.246995 4123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60355e4477ed4484960c7befa8fc2433, State: Initialized, Role: FOLLOWER
I20260808 15:48:58.247615 4123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60355e4477ed4484960c7befa8fc2433" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40955 } }
I20260808 15:48:58.247833 4123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:58.247937 4123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:58.248070 4123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:58.250458 4123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60355e4477ed4484960c7befa8fc2433" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40955 } }
I20260808 15:48:58.251009 4123 leader_election.cc:302] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60355e4477ed4484960c7befa8fc2433; no voters:
I20260808 15:48:58.251574 4123 leader_election.cc:288] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:58.251762 4133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:58.252487 4133 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [term 1 LEADER]: Becoming Leader. State: Replica: 60355e4477ed4484960c7befa8fc2433, State: Running, Role: LEADER
I20260808 15:48:58.253192 4133 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60355e4477ed4484960c7befa8fc2433" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40955 } }
I20260808 15:48:58.254062 4123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:58.256675 4134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60355e4477ed4484960c7befa8fc2433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60355e4477ed4484960c7befa8fc2433" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40955 } } }
I20260808 15:48:58.256633 4135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60355e4477ed4484960c7befa8fc2433. Latest consensus state: current_term: 1 leader_uuid: "60355e4477ed4484960c7befa8fc2433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60355e4477ed4484960c7befa8fc2433" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40955 } } }
I20260808 15:48:58.257146 4134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:58.257231 4135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:58.259454 4144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:58.263942 4144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:58.273411 4144 catalog_manager.cc:1347] Generated new cluster ID: f2b6671fef2541a38ca6853a3f3145ec
I20260808 15:48:58.273546 4144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:58.288810 4144 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:58.290098 4144 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:58.297417 4144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60355e4477ed4484960c7befa8fc2433: Generated new TSK 0
I20260808 15:48:58.298292 4144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:48:58.328480 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:58.319382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54406 (local address 127.27.84.254:40955)
0808 15:48:58.319597 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:58.319604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:58.320183 (+ 579us) server_negotiation.cc:407] Connection header received
0808 15:48:58.320412 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:58.320419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:58.320516 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:58.320712 (+ 196us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:48:58.321227 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.322392 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.323847 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.324571 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.324720 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:58.325255 (+ 535us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:48:58.327761 (+ 2506us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 15:48:58.328126 (+ 365us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 15:48:58.328620 4128 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54406: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260808 15:48:58.330189 4173 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:58.319361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:40955 (local address 127.0.0.1:54406)
0808 15:48:58.320080 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:58.320097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:58.320222 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:58.320720 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:58.320724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:58.320734 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:48:58.321069 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:58.321080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.322541 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.322545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:58.323694 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:58.323707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.324780 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.324784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:58.324922 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:58.324956 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:48:58.327993 (+ 3037us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0808 15:48:58.329742 (+ 1749us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.84.254:40955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42471/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":1434,"thread_start_us":371,"threads_started":1}
I20260808 15:48:58.331930 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 4022
I20260808 15:48:58.333645 4117 generic_service.cc:196] Checking for leaks (request via RPC)
E20260808 15:48:58.341185 4174 webserver.cc:533] Webserver: sslize failed:
E20260808 15:48:58.341284 4174 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260808 15:48:58.425482 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 4022
2026-08-08T15:48:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (780 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-08T15:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:48:58Z Disabled control of system clock
I20260808 15:48:58.524497 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36551
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36551
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/testchainca.cert with env {}
W20260808 15:48:58.868616 4180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:48:58.868963 4180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:48:58.869055 4180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:48:58.876389 4180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:48:58.876513 4180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:48:58.876554 4180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:48:58.876616 4180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:48:58.876662 4180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260808 15:48:58.876763 4180 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260808 15:48:58.876859 4180 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260808 15:48:58.887090 4180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:48:58.888417 4180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:48:58.889765 4180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:48:58.902076 4185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:58.901997 4188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:48:58.902050 4190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:48:58.902984 4180 server_base.cc:1034] running on GCE node
I20260808 15:48:58.903839 4180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:48:58.905409 4180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:48:58.906706 4180 hybrid_clock.cc:648] HybridClock initialized: now 1786204138906656 us; error 61 us; skew 500 ppm
I20260808 15:48:58.907101 4180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:48:58.909570 4180 webserver.cc:460] Webserver started at http://127.27.84.254:38735/ using document root <none> and password file <none>
I20260808 15:48:58.910347 4180 fs_manager.cc:359] Metadata directory not provided
I20260808 15:48:58.910490 4180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:48:58.910890 4180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:48:58.913928 4180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "f3474158110c4c6d964e94b9fb637c43"
format_stamp: "Formatted at 2026-08-08 15:48:58 on dist-test-slave-twwt"
server_key: "43edf5906972dbb42f28f7d9f1494871"
server_key_iv: "b417725d0b2a00383da031e0fbf9848e"
server_key_version: "encryptionkey@0"
I20260808 15:48:58.914862 4180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "f3474158110c4c6d964e94b9fb637c43"
format_stamp: "Formatted at 2026-08-08 15:48:58 on dist-test-slave-twwt"
server_key: "43edf5906972dbb42f28f7d9f1494871"
server_key_iv: "b417725d0b2a00383da031e0fbf9848e"
server_key_version: "encryptionkey@0"
I20260808 15:48:58.920746 4180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:48:58.924465 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:48:58.925882 4180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:48:58.926076 4180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "f3474158110c4c6d964e94b9fb637c43"
format_stamp: "Formatted at 2026-08-08 15:48:58 on dist-test-slave-twwt"
server_key: "43edf5906972dbb42f28f7d9f1494871"
server_key_iv: "b417725d0b2a00383da031e0fbf9848e"
server_key_version: "encryptionkey@0"
I20260808 15:48:58.926297 4180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:48:58.942205 4180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:48:58.943166 4180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:48:58.943478 4180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:48:58.962407 4180 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:36551
I20260808 15:48:58.962420 4279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:36551 every 8 connection(s)
I20260808 15:48:58.963852 4180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:48:58.967857 4282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:48:58.968736 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 4180
I20260808 15:48:58.968955 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:48:58.969676 27987 external_mini_cluster.cc:1428] Setting key 69c7dfba4358f19e0502ddf3db63625b
I20260808 15:48:58.983731 4282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43: Bootstrap starting.
I20260808 15:48:58.986089 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:58.971778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.27.84.254:36551)
0808 15:48:58.972964 (+ 1186us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:58.972980 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:58.973047 (+ 67us) server_negotiation.cc:407] Connection header received
0808 15:48:58.975059 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:58.975091 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:58.975793 (+ 702us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:58.976304 (+ 511us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:48:58.976862 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.978818 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.980291 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:58.981444 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:58.981625 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:58.983001 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:48:58.983013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:48:58.983047 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:48:58.983836 (+ 789us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:48:58.983955 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:48:58.983985 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:48:58.984109 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:48:58.984357 (+ 248us) server_negotiation.cc:299] Negotiation successful
0808 15:48:58.984703 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":329,"threads_started":1}
I20260808 15:48:58.987824 4282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43: Neither blocks nor log segments found. Creating new log.
I20260808 15:48:58.989041 4282 log.cc:824] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43: Log is configured to *not* fsync() on all Append() calls
I20260808 15:48:58.990612 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:48:58.992388 4282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43: No bootstrap required, opened a new log
I20260808 15:48:59.001133 4282 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3474158110c4c6d964e94b9fb637c43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36551 } }
I20260808 15:48:59.001700 4282 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:48:59.001817 4282 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3474158110c4c6d964e94b9fb637c43, State: Initialized, Role: FOLLOWER
I20260808 15:48:59.002666 4282 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3474158110c4c6d964e94b9fb637c43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36551 } }
I20260808 15:48:59.003038 4282 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:48:59.003185 4282 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:48:59.003400 4282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:48:59.006693 4282 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3474158110c4c6d964e94b9fb637c43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36551 } }
I20260808 15:48:59.007323 4282 leader_election.cc:302] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3474158110c4c6d964e94b9fb637c43; no voters:
I20260808 15:48:59.011034 4282 leader_election.cc:288] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:48:59.011179 4287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:48:59.012435 4287 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [term 1 LEADER]: Becoming Leader. State: Replica: f3474158110c4c6d964e94b9fb637c43, State: Running, Role: LEADER
I20260808 15:48:59.013134 4287 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3474158110c4c6d964e94b9fb637c43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36551 } }
I20260808 15:48:59.013597 4282 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:48:59.016562 4288 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3474158110c4c6d964e94b9fb637c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3474158110c4c6d964e94b9fb637c43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36551 } } }
I20260808 15:48:59.016682 4289 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3474158110c4c6d964e94b9fb637c43. Latest consensus state: current_term: 1 leader_uuid: "f3474158110c4c6d964e94b9fb637c43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3474158110c4c6d964e94b9fb637c43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36551 } } }
I20260808 15:48:59.017138 4289 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:59.017007 4288 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43 [sys.catalog]: This master's current role is: LEADER
I20260808 15:48:59.019188 4296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:48:59.024791 4296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:48:59.034817 4296 catalog_manager.cc:1347] Generated new cluster ID: 93bca5431f7f48aebd61eaed7f4deb04
I20260808 15:48:59.034974 4296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:48:59.046883 4296 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:48:59.048182 4296 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:48:59.070643 4296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3474158110c4c6d964e94b9fb637c43: Generated new TSK 0
I20260808 15:48:59.071537 4296 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:48:59.117777 4285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:48:59.095924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.27.84.254:36551)
0808 15:48:59.096118 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:48:59.096124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:48:59.096704 (+ 580us) server_negotiation.cc:407] Connection header received
0808 15:48:59.096974 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:48:59.096983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:48:59.097052 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:48:59.097181 (+ 129us) server_negotiation.cc:226] Negotiated authn=JWT
0808 15:48:59.097722 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:59.098442 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:59.100552 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:48:59.101046 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:48:59.101141 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:59.101699 (+ 558us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0808 15:48:59.117216 (+ 15517us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 15:48:59.117539 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260808 15:48:59.117973 4285 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41450: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260808 15:48:59.118292 4319 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:48:59.095836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36551 (local address 127.0.0.1:41450)
0808 15:48:59.096582 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0808 15:48:59.096604 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:48:59.096767 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:48:59.097229 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:48:59.097237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:48:59.097255 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0808 15:48:59.097563 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:59.097576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:59.098598 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:59.098624 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:59.100405 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:48:59.100422 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:48:59.101221 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:48:59.101228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:48:59.101414 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:48:59.101461 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0808 15:48:59.117511 (+ 16050us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0808 15:48:59.117841 (+ 330us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.84.254:36551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":421,"threads_started":1}
I20260808 15:48:59.119360 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 4180
I20260808 15:48:59.120638 4273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:48:59.216925 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 4180
2026-08-08T15:48:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (776 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (440 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (442 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:00 dist-test-slave-twwt krb5kdc[4341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:00 dist-test-slave-twwt krb5kdc[4341](info): set up 2 sockets
Aug 08 15:49:00 dist-test-slave-twwt krb5kdc[4341](info): commencing operation
krb5kdc: starting...
W20260808 15:49:01.469810 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.002s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:01 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:49:01.742735 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:37603
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:37603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:02.103935 4361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:02.104243 4361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:02.104417 4361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:02.111855 4361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:02.111999 4361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:02.112103 4361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:02.112201 4361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:02.112286 4361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:02.122393 4361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:37603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:37603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:02.123831 4361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:02.125240 4361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:02.137852 4375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:02.137913 4370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:02.137953 4369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:02.138695 4361 server_base.cc:1034] running on GCE node
I20260808 15:49:02.139623 4361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:02.141091 4361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:02.142305 4361 hybrid_clock.cc:648] HybridClock initialized: now 1786204142142267 us; error 29 us; skew 500 ppm
I20260808 15:49:02.142714 4361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:02 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204142, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:02.151598 4361 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:49:02.153278 4361 webserver.cc:460] Webserver started at http://127.27.84.254:40573/ using document root <none> and password file <none>
I20260808 15:49:02.154068 4361 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:02.154229 4361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:02.154587 4361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:02.157913 4361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "1af94487c9b646d6b0682cb068fb3a0b"
format_stamp: "Formatted at 2026-08-08 15:49:02 on dist-test-slave-twwt"
server_key: "135842319f23b21aed3093203217204f"
server_key_iv: "1b5da4f2b85290fabef5ccd1fdaad6b6"
server_key_version: "encryptionkey@0"
I20260808 15:49:02.158882 4361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "1af94487c9b646d6b0682cb068fb3a0b"
format_stamp: "Formatted at 2026-08-08 15:49:02 on dist-test-slave-twwt"
server_key: "135842319f23b21aed3093203217204f"
server_key_iv: "1b5da4f2b85290fabef5ccd1fdaad6b6"
server_key_version: "encryptionkey@0"
I20260808 15:49:02.164556 4361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:02.168310 4385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:02.169727 4361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:49:02.169942 4361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "1af94487c9b646d6b0682cb068fb3a0b"
format_stamp: "Formatted at 2026-08-08 15:49:02 on dist-test-slave-twwt"
server_key: "135842319f23b21aed3093203217204f"
server_key_iv: "1b5da4f2b85290fabef5ccd1fdaad6b6"
server_key_version: "encryptionkey@0"
I20260808 15:49:02.170166 4361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:02.195557 4361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:02.200354 4361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:02.200685 4361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:02.220713 4361 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:37603
I20260808 15:49:02.220737 4473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:37603 every 8 connection(s)
I20260808 15:49:02.222275 4361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:02.226423 4474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:02.231019 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 4361
I20260808 15:49:02.231268 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:02.231956 27987 external_mini_cluster.cc:1428] Setting key 3972681bb5099830c71ab90a183d0a65
I20260808 15:49:02.240152 4474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b: Bootstrap starting.
I20260808 15:49:02.244320 4474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:02.245796 4474 log.cc:824] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:49:02 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:02.249590 4474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b: No bootstrap required, opened a new log
I20260808 15:49:02.260207 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:02.234290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.27.84.254:37603)
0808 15:49:02.235290 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:02.235309 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:02.235374 (+ 65us) server_negotiation.cc:407] Connection header received
0808 15:49:02.238085 (+ 2711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:02.238121 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:02.238759 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:02.239268 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:02.240846 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:02.243008 (+ 2162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:02.244472 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:02.245588 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:02.245800 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:02.250757 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:02.250769 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:02.250792 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:02.250850 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:02.255142 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:02.255944 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:02.255953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:02.255962 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:02.256191 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:02.256631 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:02.256636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:02.256640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:02.257431 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:02.257764 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:02.258081 (+ 317us) server_negotiation.cc:299] Negotiation successful
0808 15:49:02.258654 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":450,"threads_started":1}
I20260808 15:49:02.260964 4474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af94487c9b646d6b0682cb068fb3a0b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 37603 } }
I20260808 15:49:02.261433 4474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:02.261559 4474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af94487c9b646d6b0682cb068fb3a0b, State: Initialized, Role: FOLLOWER
I20260808 15:49:02.262279 4474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af94487c9b646d6b0682cb068fb3a0b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 37603 } }
I20260808 15:49:02.262549 4474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:02.262727 4474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:02.262930 4474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:02.265401 4474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af94487c9b646d6b0682cb068fb3a0b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 37603 } }
I20260808 15:49:02.265934 4474 leader_election.cc:302] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af94487c9b646d6b0682cb068fb3a0b; no voters:
I20260808 15:49:02.266784 4474 leader_election.cc:288] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:02.266902 4480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:02.267696 4480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [term 1 LEADER]: Becoming Leader. State: Replica: 1af94487c9b646d6b0682cb068fb3a0b, State: Running, Role: LEADER
I20260808 15:49:02.268221 4480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af94487c9b646d6b0682cb068fb3a0b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 37603 } }
I20260808 15:49:02.269008 4474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:02.271490 4482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af94487c9b646d6b0682cb068fb3a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af94487c9b646d6b0682cb068fb3a0b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 37603 } } }
I20260808 15:49:02.271839 4482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:02.274821 4483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af94487c9b646d6b0682cb068fb3a0b. Latest consensus state: current_term: 1 leader_uuid: "1af94487c9b646d6b0682cb068fb3a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af94487c9b646d6b0682cb068fb3a0b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 37603 } } }
I20260808 15:49:02.275528 4483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:02.278330 4491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:02.284288 4491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:02.294168 4491 catalog_manager.cc:1347] Generated new cluster ID: e0fa1b0d27e64711a656d06ee7d312fc
I20260808 15:49:02.294333 4491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:02.310930 4491 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:02.312415 4491 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:02.327490 4491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af94487c9b646d6b0682cb068fb3a0b: Generated new TSK 0
I20260808 15:49:02.328418 4491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:49:02.402375 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:37603
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:02.756870 4513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:02.757153 4513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:02.757314 4513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:02.764881 4513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:02.764998 4513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:02.765179 4513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:02.775717 4513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:37603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:02.777086 4513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:02.778688 4513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:02.793888 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:02.794071 4522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:02.794471 4527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:02.795305 4513 server_base.cc:1034] running on GCE node
I20260808 15:49:02.795990 4513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:02.797035 4513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:02.798274 4513 hybrid_clock.cc:648] HybridClock initialized: now 1786204142798219 us; error 48 us; skew 500 ppm
I20260808 15:49:02.798681 4513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:02 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204142, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:02.802002 4513 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:49:02.803661 4513 webserver.cc:460] Webserver started at http://127.27.84.193:43977/ using document root <none> and password file <none>
I20260808 15:49:02.804406 4513 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:02.804570 4513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:02.804924 4513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:02.808123 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "95472b5119854077a04f9fec74055d39"
format_stamp: "Formatted at 2026-08-08 15:49:02 on dist-test-slave-twwt"
server_key: "df5ee220c488c32a62f080050eac5fc4"
server_key_iv: "4136cb0717aa5960105bb1c181288542"
server_key_version: "encryptionkey@0"
I20260808 15:49:02.808970 4513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "95472b5119854077a04f9fec74055d39"
format_stamp: "Formatted at 2026-08-08 15:49:02 on dist-test-slave-twwt"
server_key: "df5ee220c488c32a62f080050eac5fc4"
server_key_iv: "4136cb0717aa5960105bb1c181288542"
server_key_version: "encryptionkey@0"
I20260808 15:49:02.814471 4513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 15:49:02.818439 4540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:02.819983 4513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:02.820178 4513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "95472b5119854077a04f9fec74055d39"
format_stamp: "Formatted at 2026-08-08 15:49:02 on dist-test-slave-twwt"
server_key: "df5ee220c488c32a62f080050eac5fc4"
server_key_iv: "4136cb0717aa5960105bb1c181288542"
server_key_version: "encryptionkey@0"
I20260808 15:49:02.820410 4513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:02.851529 4513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:02.855918 4513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:02.856240 4513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:02.857308 4513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:02.858762 4513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:02.858871 4513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:02.859028 4513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:02.859114 4513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:02.894331 4513 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:33417
I20260808 15:49:02.894379 4711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:33417 every 8 connection(s)
I20260808 15:49:02.895758 4513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:02.900914 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 4513
I20260808 15:49:02.901140 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:02.901798 27987 external_mini_cluster.cc:1428] Setting key f574c80aeea2e90048daaa2f248675ee
Aug 08 15:49:02 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204142, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:02.916465 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:02.898432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:49331 (local address 127.27.84.254:37603)
0808 15:49:02.898930 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:02.898935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:02.899879 (+ 944us) server_negotiation.cc:407] Connection header received
0808 15:49:02.901881 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:02.901895 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:02.901996 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:02.902123 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:02.904002 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:02.904706 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:02.906329 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:02.906792 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:02.906934 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:02.910509 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:02.910514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:02.910518 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:02.910552 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:02.913250 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:02.914008 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:02.914015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:02.914019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:02.914149 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:02.914634 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:02.914639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:02.914642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:02.914938 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:02.915102 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:02.915827 (+ 725us) server_negotiation.cc:299] Negotiation successful
0808 15:49:02.916086 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":273,"threads_started":1}
I20260808 15:49:02.917506 4715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:02.898977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:37603 (local address 127.27.84.193:49331)
0808 15:49:02.899711 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:02.899756 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:02.901593 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:02.902379 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:02.902394 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:02.903129 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:02.903828 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:02.903849 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:02.904897 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:02.904901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:02.906176 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:02.906189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:02.907041 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:02.907049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:02.907302 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:02.908017 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:02.908042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:02.910267 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:02.913420 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:02.913430 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:02.913448 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:02.913879 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:02.914268 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:02.914273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:02.914277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:02.914491 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:02.915076 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:02.915084 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:02.915446 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 15:49:02.915728 (+ 282us) client_negotiation.cc:240] Negotiation successful
0808 15:49:02.916279 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":313,"threads_started":1}
I20260808 15:49:02.919260 4712 heartbeater.cc:347] Connected to a master server at 127.27.84.254:37603
I20260808 15:49:02.919592 4712 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:02.920343 4712 heartbeater.cc:511] Master 127.27.84.254:37603 requested a full tablet report, sending...
I20260808 15:49:02.922915 4416 ts_manager.cc:194] Registered new tserver with Master: 95472b5119854077a04f9fec74055d39 (127.27.84.193:33417)
I20260808 15:49:02.925300 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:49331
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:49:02.962495 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:37603
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:03.335386 4719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:03.335736 4719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:03.335906 4719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:03.343372 4719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:03.343534 4719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:03.343712 4719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:03.353855 4719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:37603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:03.355386 4719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:03.356747 4719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:03.370283 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:03.370388 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:03.370347 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:03.371634 4719 server_base.cc:1034] running on GCE node
I20260808 15:49:03.372279 4719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:03.373282 4719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:03.374492 4719 hybrid_clock.cc:648] HybridClock initialized: now 1786204143374428 us; error 55 us; skew 500 ppm
I20260808 15:49:03.374881 4719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:03 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:03.378260 4719 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:49:03.379978 4719 webserver.cc:460] Webserver started at http://127.27.84.194:46825/ using document root <none> and password file <none>
I20260808 15:49:03.380748 4719 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:03.380858 4719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:03.381127 4719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:03.384310 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "3c6e4f333a2640a28ce0513f11f6e7d9"
format_stamp: "Formatted at 2026-08-08 15:49:03 on dist-test-slave-twwt"
server_key: "032a80bd23d7ede066431275f608ce65"
server_key_iv: "b23885f4481fad09a5f5d25f52a39f64"
server_key_version: "encryptionkey@0"
I20260808 15:49:03.385103 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "3c6e4f333a2640a28ce0513f11f6e7d9"
format_stamp: "Formatted at 2026-08-08 15:49:03 on dist-test-slave-twwt"
server_key: "032a80bd23d7ede066431275f608ce65"
server_key_iv: "b23885f4481fad09a5f5d25f52a39f64"
server_key_version: "encryptionkey@0"
I20260808 15:49:03.390584 4719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260808 15:49:03.394164 4741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:03.395654 4719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:49:03.395938 4719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "3c6e4f333a2640a28ce0513f11f6e7d9"
format_stamp: "Formatted at 2026-08-08 15:49:03 on dist-test-slave-twwt"
server_key: "032a80bd23d7ede066431275f608ce65"
server_key_iv: "b23885f4481fad09a5f5d25f52a39f64"
server_key_version: "encryptionkey@0"
I20260808 15:49:03.396171 4719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:03.417865 4719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:03.422554 4719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:03.422928 4719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:03.424013 4719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:03.425468 4719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:03.425583 4719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:03.425729 4719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:03.425808 4719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:03.460620 4719 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:33275
I20260808 15:49:03.460672 4922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:33275 every 8 connection(s)
I20260808 15:49:03.461966 4719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:03.472009 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 4719
I20260808 15:49:03.472196 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:03.472656 27987 external_mini_cluster.cc:1428] Setting key 2900aa9709fdc7ca4c69385fdc22e44f
Aug 08 15:49:03 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:03.483148 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:03.464611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:48495 (local address 127.27.84.254:37603)
0808 15:49:03.465059 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:03.465063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:03.466141 (+ 1078us) server_negotiation.cc:407] Connection header received
0808 15:49:03.468036 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:03.468040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:03.468117 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:03.468220 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:03.469844 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:03.470775 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:03.472371 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:03.472902 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:03.473043 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:03.476970 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:03.476975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:03.476978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:03.477011 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:03.479941 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:03.480689 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:03.480701 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:03.480703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:03.480837 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:03.481312 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:03.481324 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:03.481326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:03.481602 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:03.481760 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:03.482477 (+ 717us) server_negotiation.cc:299] Negotiation successful
0808 15:49:03.482738 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":233,"threads_started":1}
I20260808 15:49:03.484200 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:03.465210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:37603 (local address 127.27.84.194:48495)
0808 15:49:03.465963 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:03.466009 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:03.467780 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:03.468449 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:03.468457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:03.469003 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:03.469596 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:03.469615 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:03.470926 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:03.470930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:03.472209 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:03.472221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:03.473131 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:03.473141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:03.473396 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:03.474253 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:03.474282 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:03.476729 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:03.480089 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:03.480098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:03.480115 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:03.480556 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:03.480964 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:03.480969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:03.480973 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:03.481185 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:03.481734 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:03.481743 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:03.482109 (+ 366us) client_negotiation.cc:769] Sending connection context
0808 15:49:03.482395 (+ 286us) client_negotiation.cc:240] Negotiation successful
0808 15:49:03.483014 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":297,"threads_started":1}
I20260808 15:49:03.485951 4923 heartbeater.cc:347] Connected to a master server at 127.27.84.254:37603
I20260808 15:49:03.486359 4923 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:03.487205 4923 heartbeater.cc:511] Master 127.27.84.254:37603 requested a full tablet report, sending...
I20260808 15:49:03.488901 4416 ts_manager.cc:194] Registered new tserver with Master: 3c6e4f333a2640a28ce0513f11f6e7d9 (127.27.84.194:33275)
I20260808 15:49:03.490180 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:48495
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:49:03.534269 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:37603
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:03.884886 4935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:03.885280 4935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:03.885470 4935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:03.892962 4935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:03.893090 4935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:03.893208 4935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:03.904001 4935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:37603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:03.905324 4935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:03.906790 4935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:03.920881 4946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:03.920972 4942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:03.921139 4943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:03.922001 4935 server_base.cc:1034] running on GCE node
I20260808 15:49:03.922711 4935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:03.923856 4935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:03.925096 4935 hybrid_clock.cc:648] HybridClock initialized: now 1786204143925046 us; error 59 us; skew 500 ppm
I20260808 15:49:03.925431 4935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:03 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:03.928483 4712 heartbeater.cc:503] Master 127.27.84.254:37603 was elected leader, sending a full tablet report...
I20260808 15:49:03.928689 4935 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:49:03.930445 4935 webserver.cc:460] Webserver started at http://127.27.84.195:37529/ using document root <none> and password file <none>
I20260808 15:49:03.931219 4935 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:03.931378 4935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:03.931701 4935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:03.934926 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "4a8f9bffa7fa4c359fa37c32dd813aa0"
format_stamp: "Formatted at 2026-08-08 15:49:03 on dist-test-slave-twwt"
server_key: "0db5f68bb2f2c11f603757d26b1f90cd"
server_key_iv: "6bbd4b695bcc75d11ee7595659a07e8f"
server_key_version: "encryptionkey@0"
I20260808 15:49:03.935765 4935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "4a8f9bffa7fa4c359fa37c32dd813aa0"
format_stamp: "Formatted at 2026-08-08 15:49:03 on dist-test-slave-twwt"
server_key: "0db5f68bb2f2c11f603757d26b1f90cd"
server_key_iv: "6bbd4b695bcc75d11ee7595659a07e8f"
server_key_version: "encryptionkey@0"
I20260808 15:49:03.941277 4935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:03.944990 4958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:03.946426 4935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:03.946694 4935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "4a8f9bffa7fa4c359fa37c32dd813aa0"
format_stamp: "Formatted at 2026-08-08 15:49:03 on dist-test-slave-twwt"
server_key: "0db5f68bb2f2c11f603757d26b1f90cd"
server_key_iv: "6bbd4b695bcc75d11ee7595659a07e8f"
server_key_version: "encryptionkey@0"
I20260808 15:49:03.946941 4935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:03.963599 4935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:03.968024 4935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:03.968341 4935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:03.969477 4935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:03.971055 4935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:03.971163 4935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:03.971315 4935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:03.971396 4935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:04.007442 4935 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:43089
I20260808 15:49:04.007486 5127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:43089 every 8 connection(s)
I20260808 15:49:04.008867 4935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:04.011852 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 4935
I20260808 15:49:04.012072 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:04.012586 27987 external_mini_cluster.cc:1428] Setting key 279fdca198d8eb354a1d7df84135bae7
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204143, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:04.035243 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.011738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:32905 (local address 127.27.84.254:37603)
0808 15:49:04.012298 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.012305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.013498 (+ 1193us) server_negotiation.cc:407] Connection header received
0808 15:49:04.016061 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.016067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.016179 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.016271 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.019255 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.020548 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.022749 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.023402 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.023509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.028890 (+ 5381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.028896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.028900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.028943 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.031945 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.032731 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.032735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.032737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.032869 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.033298 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.033300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.033302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.033578 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.033757 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.034540 (+ 783us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.034803 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":263,"threads_started":1}
I20260808 15:49:04.036280 5134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.012365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:37603 (local address 127.27.84.195:32905)
0808 15:49:04.013317 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:04.013359 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:04.015714 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:04.016966 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:04.016980 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:04.017946 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:04.019004 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:04.019034 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.020741 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.020747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:04.022541 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:04.022560 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.023961 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.023969 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:04.024285 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.025251 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:04.025278 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:04.028605 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:04.032112 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:04.032120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:04.032136 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:04.032603 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:04.032982 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:04.032985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:04.032987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:04.033192 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:04.033730 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:04.033740 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:04.034145 (+ 405us) client_negotiation.cc:769] Sending connection context
0808 15:49:04.034446 (+ 301us) client_negotiation.cc:240] Negotiation successful
0808 15:49:04.035075 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":390,"threads_started":1}
I20260808 15:49:04.037784 5129 heartbeater.cc:347] Connected to a master server at 127.27.84.254:37603
I20260808 15:49:04.038113 5129 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:04.038973 5129 heartbeater.cc:511] Master 127.27.84.254:37603 requested a full tablet report, sending...
I20260808 15:49:04.040670 4416 ts_manager.cc:194] Registered new tserver with Master: 4a8f9bffa7fa4c359fa37c32dd813aa0 (127.27.84.195:43089)
I20260808 15:49:04.041800 4416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:32905
I20260808 15:49:04.041919 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/test-admin.keytab.
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.076009 27987 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260808 15:49:04.081398 5140 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:04.078833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48286 (local address 127.27.84.193:33417)
0808 15:49:04.079626 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.079633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.079661 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:49:04.079725 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.079732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.079966 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.080119 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.080983 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48286: BlockingRecv error: recv got EOF from 127.0.0.1:48286 (error 108)
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":469,"threads_started":1}
W20260808 15:49:04.081504 27987 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.083624 27987 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.085011 27987 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:49:04.101466 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.086815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48294 (local address 127.27.84.193:33417)
0808 15:49:04.087010 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.087017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.087168 (+ 151us) server_negotiation.cc:407] Connection header received
0808 15:49:04.087319 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.087322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.087392 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.087517 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.088900 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.089947 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.091231 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.091788 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.091891 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.095748 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.095762 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.095778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.095822 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.098560 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.099283 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.099290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.099308 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.099495 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.099835 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.099839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.099841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.100207 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.100448 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.100835 (+ 387us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.101165 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:49:04.107887 5140 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:04.105808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.27.84.193:33417)
0808 15:49:04.106038 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.106047 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.106353 (+ 306us) server_negotiation.cc:407] Connection header received
0808 15:49:04.106555 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.106559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.106677 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.106783 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.107660 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260808 15:49:04.108530 27987 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.110117 27987 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.111480 27987 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:49:04.126597 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.112790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.27.84.193:33417)
0808 15:49:04.113027 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.113033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.113249 (+ 216us) server_negotiation.cc:407] Connection header received
0808 15:49:04.113388 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.113391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.113452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.113534 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.114769 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.115473 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.116655 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.117074 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.117181 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.120973 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.120978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.120982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.121018 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.123879 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.124608 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.124615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.124619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.124757 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.125217 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.125220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.125222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.125508 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.125814 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.126082 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.126305 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260808 15:49:04.131071 5140 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:04.129456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.27.84.193:33417)
0808 15:49:04.129644 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.129649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.129685 (+ 36us) server_negotiation.cc:407] Connection header received
0808 15:49:04.129855 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.129859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.129927 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.130007 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.130792 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48322: BlockingRecv error: recv got EOF from 127.0.0.1:48322 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 15:49:04.131129 27987 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786204089698720-27987-0/krb5kdc/krb5cc' of type FILE
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.132776 27987 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:04.134359 27987 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:49:04.149165 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.135806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.27.84.193:33417)
0808 15:49:04.136037 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.136043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.136279 (+ 236us) server_negotiation.cc:407] Connection header received
0808 15:49:04.136389 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.136396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.136481 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.136592 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.137564 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.138299 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.139599 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.140056 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.140156 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.143831 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.143836 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.143840 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.143874 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.146521 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.147192 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.147196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.147198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.147345 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.147735 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.147738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.147740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.148016 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.148341 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.148630 (+ 289us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.148846 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:49:04.151022 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 4513
I20260808 15:49:04.163043 5140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.151825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.27.84.193:33417)
0808 15:49:04.152060 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.152065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.152342 (+ 277us) server_negotiation.cc:407] Connection header received
0808 15:49:04.152469 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.152473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.152527 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.152603 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.153522 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.154175 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.155439 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.155830 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.155941 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.157824 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.157829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.157832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.157863 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.160749 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.161344 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.161347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.161349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.161481 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.161802 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.161806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.161807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.162073 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.162276 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.162527 (+ 251us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.162761 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:49:04.163703 4707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:04.250386 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 4513
I20260808 15:49:04.277016 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 4719
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:49:04.293093 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.277948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55008 (local address 127.27.84.194:33275)
0808 15:49:04.278526 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.278537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.278574 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:49:04.278700 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.278708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.278940 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.279129 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.280197 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.281198 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.282409 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.283145 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.283249 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.287440 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.287454 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.287468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.287503 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.290390 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.291155 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.291163 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.291165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.291293 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.291644 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.291647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.291649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.292025 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.292257 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.292539 (+ 282us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.292763 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":325,"threads_started":1}
I20260808 15:49:04.294438 4915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:04.381078 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 4719
I20260808 15:49:04.405098 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 4935
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[4341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:49:04.421285 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:04.406220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.27.84.195:43089)
0808 15:49:04.406971 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:04.406978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:04.407008 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:49:04.407071 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:04.407078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:04.407299 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:04.407444 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:04.408550 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.409558 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.410880 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:04.411451 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:04.411550 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:04.415667 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:04.415680 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:04.415695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:04.415729 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:04.418490 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.419347 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.419355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.419357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.419493 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:04.419840 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:04.419843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:04.419845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:04.420196 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:04.420447 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:04.420680 (+ 233us) server_negotiation.cc:299] Negotiation successful
0808 15:49:04.420969 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":424,"threads_started":1}
I20260808 15:49:04.422818 5121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:04.506359 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 4935
I20260808 15:49:04.529930 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 4361
I20260808 15:49:04.531155 4468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:04.637038 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 4361
2026-08-08T15:49:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4541 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[5165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[5165](info): set up 2 sockets
Aug 08 15:49:04 dist-test-slave-twwt krb5kdc[5165](info): commencing operation
krb5kdc: starting...
W20260808 15:49:05.996668 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:06 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab.
Entry for principal oryx/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab.
I20260808 15:49:06.278748 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:39485
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab
--principal=oryx/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:39485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:06.632525 5188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:06.632947 5188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:06.633119 5188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:06.640702 5188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:06.640863 5188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:06.640969 5188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:06.641072 5188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:06.641149 5188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:06.651204 5188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:39485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab
--principal=oryx/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:06.652578 5188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:06.653976 5188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:06.666337 5198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:06.666363 5195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:06.666330 5200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:06.667301 5188 server_base.cc:1034] running on GCE node
I20260808 15:49:06.668078 5188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:06.669507 5188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:06.670723 5188 hybrid_clock.cc:648] HybridClock initialized: now 1786204146670657 us; error 45 us; skew 500 ppm
I20260808 15:49:06.671104 5188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:06 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:06.674417 5188 init.cc:377] Logged in from keytab as oryx/127.27.84.254@KRBTEST.COM (short username oryx)
I20260808 15:49:06.676144 5188 webserver.cc:460] Webserver started at http://127.27.84.254:35663/ using document root <none> and password file <none>
I20260808 15:49:06.676870 5188 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:06.677023 5188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:06.677351 5188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:06.680505 5188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "b173194b048f444294d077c949db5fc6"
format_stamp: "Formatted at 2026-08-08 15:49:06 on dist-test-slave-twwt"
server_key: "fafe05a83cbb98610e354e97f3ab3445"
server_key_iv: "8ab6e5867981c02d10727eef5929386a"
server_key_version: "encryptionkey@0"
I20260808 15:49:06.681346 5188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "b173194b048f444294d077c949db5fc6"
format_stamp: "Formatted at 2026-08-08 15:49:06 on dist-test-slave-twwt"
server_key: "fafe05a83cbb98610e354e97f3ab3445"
server_key_iv: "8ab6e5867981c02d10727eef5929386a"
server_key_version: "encryptionkey@0"
I20260808 15:49:06.687326 5188 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260808 15:49:06.691466 5208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:06.693131 5188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260808 15:49:06.693342 5188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "b173194b048f444294d077c949db5fc6"
format_stamp: "Formatted at 2026-08-08 15:49:06 on dist-test-slave-twwt"
server_key: "fafe05a83cbb98610e354e97f3ab3445"
server_key_iv: "8ab6e5867981c02d10727eef5929386a"
server_key_version: "encryptionkey@0"
I20260808 15:49:06.693568 5188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:06.709616 5188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:06.714289 5188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:06.714602 5188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:06.734056 5188 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:39485
I20260808 15:49:06.734081 5286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:39485 every 8 connection(s)
I20260808 15:49:06.735608 5188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:06.737464 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 5188
I20260808 15:49:06.737681 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:06.738633 27987 external_mini_cluster.cc:1428] Setting key d0d42f821691b24b241f64bdd9811e6f
I20260808 15:49:06.740556 5287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:06.752223 5287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Bootstrap starting.
Aug 08 15:49:06 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:06.756073 5287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:06.757195 5287 log.cc:824] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:06.760773 5287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: No bootstrap required, opened a new log
I20260808 15:49:06.761778 5292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:06.741077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.27.84.254:39485)
0808 15:49:06.741860 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:06.741873 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:06.741915 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:49:06.743624 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:06.743647 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:06.744053 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:06.744484 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:06.746021 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:06.747413 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:06.748616 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:06.749316 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:06.749535 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:06.753929 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:06.753942 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:06.753956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:06.753998 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:06.757328 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:06.758222 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:06.758231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:06.758239 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:06.758501 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:06.758947 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:06.758950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:06.758952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:06.759476 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:06.759702 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:06.760220 (+ 518us) server_negotiation.cc:299] Negotiation successful
0808 15:49:06.760639 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":356,"threads_started":1}
I20260808 15:49:06.768715 5287 raft_consensus.cc:348] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } }
I20260808 15:49:06.769124 5287 raft_consensus.cc:374] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:06.769227 5287 raft_consensus.cc:729] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b173194b048f444294d077c949db5fc6, State: Initialized, Role: FOLLOWER
I20260808 15:49:06.769804 5287 consensus_queue.cc:260] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } }
I20260808 15:49:06.770054 5287 raft_consensus.cc:388] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:06.770211 5287 raft_consensus.cc:482] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:06.770411 5287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:06.772961 5287 raft_consensus.cc:504] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } }
I20260808 15:49:06.773499 5287 leader_election.cc:302] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b173194b048f444294d077c949db5fc6; no voters:
I20260808 15:49:06.774125 5287 leader_election.cc:288] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:06.774276 5295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:06.775080 5295 raft_consensus.cc:686] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 LEADER]: Becoming Leader. State: Replica: b173194b048f444294d077c949db5fc6, State: Running, Role: LEADER
I20260808 15:49:06.775662 5295 consensus_queue.cc:237] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } }
I20260808 15:49:06.776294 5287 sys_catalog.cc:564] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:06.779111 5298 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b173194b048f444294d077c949db5fc6. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } }
I20260808 15:49:06.779531 5298 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:06.779320 5296 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } }
I20260808 15:49:06.779731 5296 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:06.783953 5306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:06.789373 5306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:06.800364 5306 catalog_manager.cc:1347] Generated new cluster ID: cf9b92ee36964699ba0615c5f9a7497b
I20260808 15:49:06.800484 5306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:06.816646 5306 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:06.817977 5306 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:06.826208 5306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Generated new TSK 0
I20260808 15:49:06.827167 5306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab.
Entry for principal oryx/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab.
I20260808 15:49:06.890115 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39485
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:07.247383 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:07.247671 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:07.247835 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:07.255295 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:07.255429 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:07.255635 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:07.266299 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:07.267707 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:07.269127 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:07.283381 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:07.283602 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:07.283331 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:07.284989 5328 server_base.cc:1034] running on GCE node
I20260808 15:49:07.285784 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:07.286881 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:07.288095 5328 hybrid_clock.cc:648] HybridClock initialized: now 1786204147288047 us; error 36 us; skew 500 ppm
I20260808 15:49:07.288476 5328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:07 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204147, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:07.300316 5328 init.cc:377] Logged in from keytab as oryx/127.27.84.193@KRBTEST.COM (short username oryx)
I20260808 15:49:07.302100 5328 webserver.cc:460] Webserver started at http://127.27.84.193:40179/ using document root <none> and password file <none>
I20260808 15:49:07.302965 5328 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:07.303128 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:07.303472 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:07.306572 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "ee5ec6fb7ca846df819e9a930eff7e5f"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "9f0f71ab679cdcb2d980993146b306f0"
server_key_iv: "85a4ed26662a2dd13d1a16e3186c5f94"
server_key_version: "encryptionkey@0"
I20260808 15:49:07.307494 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "ee5ec6fb7ca846df819e9a930eff7e5f"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "9f0f71ab679cdcb2d980993146b306f0"
server_key_iv: "85a4ed26662a2dd13d1a16e3186c5f94"
server_key_version: "encryptionkey@0"
I20260808 15:49:07.313014 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:07.316586 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:07.317962 5328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:07.318146 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "ee5ec6fb7ca846df819e9a930eff7e5f"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "9f0f71ab679cdcb2d980993146b306f0"
server_key_iv: "85a4ed26662a2dd13d1a16e3186c5f94"
server_key_version: "encryptionkey@0"
I20260808 15:49:07.318351 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:07.342439 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:07.347052 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:07.347358 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:07.348426 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:07.349850 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:07.349958 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:07.350109 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:07.350185 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:07.386320 5328 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:33515
I20260808 15:49:07.386387 5534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:33515 every 8 connection(s)
I20260808 15:49:07.387758 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:07.390992 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 5328
I20260808 15:49:07.391251 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:07.392261 27987 external_mini_cluster.cc:1428] Setting key b5255b814db6f698f3aab31b6c992cda
Aug 08 15:49:07 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204147, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:07.414078 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:07.390599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:44159 (local address 127.27.84.254:39485)
0808 15:49:07.391238 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:07.391248 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:07.392511 (+ 1263us) server_negotiation.cc:407] Connection header received
0808 15:49:07.395197 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:07.395202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:07.395270 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:07.395380 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:07.398057 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.398968 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.401567 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.401964 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.402067 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:07.406641 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:07.406651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:07.406658 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:07.406708 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:07.410495 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:07.411325 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:07.411334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:07.411341 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:07.411498 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:07.411961 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:07.411968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:07.411974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:07.412389 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:07.412569 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:07.413318 (+ 749us) server_negotiation.cc:299] Negotiation successful
0808 15:49:07.413592 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":267,"threads_started":1}
I20260808 15:49:07.414839 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:07.391266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:44159)
0808 15:49:07.392277 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:07.392360 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:07.394829 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:07.395718 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:07.395734 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:07.396680 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:07.397801 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:07.397833 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.399144 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.399152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:07.401392 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:07.401414 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.402211 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.402222 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:07.402574 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:07.403410 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:07.403448 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:07.406328 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:07.410706 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:07.410713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:07.410728 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:07.411183 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:07.411633 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:07.411636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:07.411638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:07.411834 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:07.412524 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:07.412531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:07.412896 (+ 365us) client_negotiation.cc:769] Sending connection context
0808 15:49:07.413136 (+ 240us) client_negotiation.cc:240] Negotiation successful
0808 15:49:07.413569 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":330,"threads_started":1}
I20260808 15:49:07.416563 5535 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39485
I20260808 15:49:07.416913 5535 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:07.417699 5535 heartbeater.cc:511] Master 127.27.84.254:39485 requested a full tablet report, sending...
I20260808 15:49:07.420020 5234 ts_manager.cc:194] Registered new tserver with Master: ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515)
I20260808 15:49:07.421643 5234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:44159
WARNING: no policy specified for oryx/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab.
Entry for principal oryx/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab.
I20260808 15:49:07.458066 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39485
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:07.818890 5544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:07.819260 5544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:07.819427 5544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:07.827224 5544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:07.827371 5544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:07.827551 5544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:07.837903 5544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:07.840129 5544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:07.841626 5544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:07.855914 5552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:07.855988 5553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:07.855949 5557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:07.857597 5544 server_base.cc:1034] running on GCE node
I20260808 15:49:07.858537 5544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:07.859623 5544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:07.860894 5544 hybrid_clock.cc:648] HybridClock initialized: now 1786204147860821 us; error 66 us; skew 500 ppm
I20260808 15:49:07.861238 5544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:07 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204147, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:07.864714 5544 init.cc:377] Logged in from keytab as oryx/127.27.84.194@KRBTEST.COM (short username oryx)
I20260808 15:49:07.866370 5544 webserver.cc:460] Webserver started at http://127.27.84.194:42325/ using document root <none> and password file <none>
I20260808 15:49:07.867188 5544 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:07.867347 5544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:07.867672 5544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:07.871047 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "d4b1753305774f8e81ce34ae635a4df3"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "b94d89f7324142f67f433dd1aa3850e9"
server_key_iv: "12191aabfb66c407922f36bf9c74daab"
server_key_version: "encryptionkey@0"
I20260808 15:49:07.871941 5544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "d4b1753305774f8e81ce34ae635a4df3"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "b94d89f7324142f67f433dd1aa3850e9"
server_key_iv: "12191aabfb66c407922f36bf9c74daab"
server_key_version: "encryptionkey@0"
I20260808 15:49:07.878602 5544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 15:49:07.882494 5564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:07.884281 5544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:07.884548 5544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "d4b1753305774f8e81ce34ae635a4df3"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "b94d89f7324142f67f433dd1aa3850e9"
server_key_iv: "12191aabfb66c407922f36bf9c74daab"
server_key_version: "encryptionkey@0"
I20260808 15:49:07.884789 5544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:07.898566 5544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:07.903230 5544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:07.903591 5544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:07.904639 5544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:07.906231 5544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:07.906352 5544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:07.906519 5544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:07.906596 5544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:07.942454 5544 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:40679
I20260808 15:49:07.942500 5737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:40679 every 8 connection(s)
I20260808 15:49:07.943925 5544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:07.947467 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 5544
I20260808 15:49:07.947662 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:07.948154 27987 external_mini_cluster.cc:1428] Setting key 9367a3dd186b68dc556917fb80127ac3
Aug 08 15:49:07 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204147, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:07.969713 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:07.946745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:40071 (local address 127.27.84.254:39485)
0808 15:49:07.947266 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:07.947274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:07.948459 (+ 1185us) server_negotiation.cc:407] Connection header received
0808 15:49:07.950910 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:07.950918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:07.951033 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:07.951148 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:07.953678 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.955181 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.957678 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.958422 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.958580 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:07.963608 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:07.963613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:07.963617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:07.963653 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:07.966308 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:07.967200 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:07.967205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:07.967208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:07.967346 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:07.967842 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:07.967852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:07.967854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:07.968122 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:07.968286 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:07.969054 (+ 768us) server_negotiation.cc:299] Negotiation successful
0808 15:49:07.969307 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":233,"threads_started":1}
I20260808 15:49:07.970878 5742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:07.947352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:40071)
0808 15:49:07.948242 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:07.948300 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:07.950494 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:07.951561 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:07.951575 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:07.952452 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:07.953442 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:07.953478 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.955400 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.955406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:07.957112 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:07.957132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:07.958749 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:07.958761 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:07.959148 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:07.960274 (+ 1126us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:07.960313 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:07.963332 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:07.966473 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:07.966484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:07.966504 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:07.967051 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:07.967471 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:07.967476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:07.967480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:07.967703 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:07.968262 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:07.968270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:07.968660 (+ 390us) client_negotiation.cc:769] Sending connection context
0808 15:49:07.968943 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 15:49:07.969566 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":399,"threads_started":1}
I20260808 15:49:07.972568 5738 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39485
I20260808 15:49:07.972929 5738 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:07.973816 5738 heartbeater.cc:511] Master 127.27.84.254:39485 requested a full tablet report, sending...
I20260808 15:49:07.975669 5234 ts_manager.cc:194] Registered new tserver with Master: d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679)
I20260808 15:49:07.977013 5234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:40071
WARNING: no policy specified for oryx/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab.
Entry for principal oryx/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab.
I20260808 15:49:08.014169 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39485
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:08.365854 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:08.366151 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:08.366321 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:08.373683 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:08.373799 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:08.374009 5747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:08.384564 5747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:08.386343 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:08.388283 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:08.403501 5756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:08.403584 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:08.403610 5759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:08.405303 5747 server_base.cc:1034] running on GCE node
I20260808 15:49:08.406092 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:08.407232 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:08.408494 5747 hybrid_clock.cc:648] HybridClock initialized: now 1786204148408406 us; error 62 us; skew 500 ppm
I20260808 15:49:08.408849 5747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:08 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204148, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:08.412365 5747 init.cc:377] Logged in from keytab as oryx/127.27.84.195@KRBTEST.COM (short username oryx)
I20260808 15:49:08.414007 5747 webserver.cc:460] Webserver started at http://127.27.84.195:37709/ using document root <none> and password file <none>
I20260808 15:49:08.414767 5747 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:08.414883 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:08.415146 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:08.418170 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "cb92e1212952499aaed5d509a19d8683"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "c7a1d110faaf90d92228c75842762e16"
server_key_iv: "554e287c72eef7d94102708f2e54cb92"
server_key_version: "encryptionkey@0"
I20260808 15:49:08.419051 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "cb92e1212952499aaed5d509a19d8683"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "c7a1d110faaf90d92228c75842762e16"
server_key_iv: "554e287c72eef7d94102708f2e54cb92"
server_key_version: "encryptionkey@0"
I20260808 15:49:08.424746 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:08.424988 5535 heartbeater.cc:503] Master 127.27.84.254:39485 was elected leader, sending a full tablet report...
I20260808 15:49:08.428886 5767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:08.430260 5747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260808 15:49:08.430409 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "cb92e1212952499aaed5d509a19d8683"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "c7a1d110faaf90d92228c75842762e16"
server_key_iv: "554e287c72eef7d94102708f2e54cb92"
server_key_version: "encryptionkey@0"
I20260808 15:49:08.430593 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:08.445113 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:08.450191 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:08.450516 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:08.451546 5747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:08.452955 5747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:08.453059 5747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:08.453219 5747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:08.453305 5747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:08.489262 5747 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:45919
I20260808 15:49:08.489281 5943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:45919 every 8 connection(s)
I20260808 15:49:08.490757 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:08.492771 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 5747
I20260808 15:49:08.492971 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:08.493602 27987 external_mini_cluster.cc:1428] Setting key ed8bfb3ad085baf30802ed72685c043c
Aug 08 15:49:08 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204148, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:08.512514 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:08.493655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:39159 (local address 127.27.84.254:39485)
0808 15:49:08.494233 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:08.494239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:08.495491 (+ 1252us) server_negotiation.cc:407] Connection header received
0808 15:49:08.497890 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:08.497894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:08.498008 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:08.498127 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:08.499874 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:08.500682 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:08.502252 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:08.502720 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:08.502821 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:08.506249 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:08.506253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:08.506257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:08.506290 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:08.509569 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:08.510353 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:08.510357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:08.510360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:08.510494 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:08.510974 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:08.510977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:08.510979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:08.511239 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:08.511363 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:08.512065 (+ 702us) server_negotiation.cc:299] Negotiation successful
0808 15:49:08.512230 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":298,"threads_started":1}
I20260808 15:49:08.513502 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:08.494352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:39159)
0808 15:49:08.495265 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:08.495364 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:08.497615 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:08.498483 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:08.498492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:08.499111 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:08.499711 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:08.499730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:08.500825 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:08.500829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:08.502113 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:08.502125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:08.502936 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:08.502942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:08.503151 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:08.503767 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:08.503790 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:08.506051 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:08.509700 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:08.509707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:08.509720 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:08.510245 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:08.510594 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:08.510597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:08.510599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:08.510874 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:08.511365 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:08.511370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:08.511695 (+ 325us) client_negotiation.cc:769] Sending connection context
0808 15:49:08.511963 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 15:49:08.512353 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":394,"threads_started":1}
I20260808 15:49:08.514952 5945 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39485
I20260808 15:49:08.515265 5945 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:08.516041 5945 heartbeater.cc:511] Master 127.27.84.254:39485 requested a full tablet report, sending...
I20260808 15:49:08.517484 5234 ts_manager.cc:194] Registered new tserver with Master: cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919)
I20260808 15:49:08.518283 5234 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:39159
I20260808 15:49:08.522219 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.84.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.253.keytab.
Entry for principal oryx/127.27.84.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.253.keytab.
I20260808 15:49:08.584471 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:41089
--webserver_interface=127.27.84.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.253.keytab
--principal=oryx/127.27.84.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:39485,127.27.84.253:41089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:08.942724 5952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:08.943094 5952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:08.943259 5952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:08.951381 5952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:08.951550 5952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:08.951664 5952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:08.951781 5952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:08.951861 5952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:08.962376 5952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:39485,127.27.84.253:41089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.253.keytab
--principal=oryx/127.27.84.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:41089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.84.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
W20260808 15:49:08.962862 5952 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.84.254:39485,127.27.84.253:41089'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260808 15:49:08.964031 5952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:08.965533 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:08.978600 5959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:08.978690 5961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:08.978690 5964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:08.979669 5952 server_base.cc:1034] running on GCE node
I20260808 15:49:08.980239 5738 heartbeater.cc:503] Master 127.27.84.254:39485 was elected leader, sending a full tablet report...
I20260808 15:49:08.980628 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:08.982447 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:08.983749 5952 hybrid_clock.cc:648] HybridClock initialized: now 1786204148983650 us; error 64 us; skew 500 ppm
I20260808 15:49:08.984175 5952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:08 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204148, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:08.987790 5952 init.cc:377] Logged in from keytab as oryx/127.27.84.253@KRBTEST.COM (short username oryx)
I20260808 15:49:08.989607 5952 webserver.cc:460] Webserver started at http://127.27.84.253:39691/ using document root <none> and password file <none>
I20260808 15:49:08.990453 5952 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:08.990597 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:08.990998 5952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:08.994263 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/instance:
uuid: "fa99b51165794de98294ac7c51b86512"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "6a461f8b0c833df2da984d9ccf9a61a9"
server_key_iv: "fb0c58697b401fa408df2723db0cd3f8"
server_key_version: "encryptionkey@0"
I20260808 15:49:08.995138 5952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal/instance:
uuid: "fa99b51165794de98294ac7c51b86512"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "6a461f8b0c833df2da984d9ccf9a61a9"
server_key_iv: "fb0c58697b401fa408df2723db0cd3f8"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.001053 5952 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260808 15:49:09.004945 5973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:09.006578 5952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:09.006867 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
uuid: "fa99b51165794de98294ac7c51b86512"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "6a461f8b0c833df2da984d9ccf9a61a9"
server_key_iv: "fb0c58697b401fa408df2723db0cd3f8"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.007126 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:09.026438 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:09.033131 5952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:09.033481 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:09.057711 5952 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.253:41089
I20260808 15:49:09.057730 6054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.253:41089 every 8 connection(s)
I20260808 15:49:09.059316 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
I20260808 15:49:09.063227 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 5952
I20260808 15:49:09.063908 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:09.073370 6056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204148, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.253@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:09.090075 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.065736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.27.84.253:41089)
0808 15:49:09.066522 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.066534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.066575 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:49:09.068941 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.068964 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.069414 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.069883 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.071586 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.072936 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.074301 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.075385 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.075581 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.081039 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.081048 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.081062 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.081115 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.084697 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.085652 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.085667 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.085681 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.085954 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.086505 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.086514 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.086519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.087344 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.087681 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.088073 (+ 392us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.088592 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":342,"threads_started":1}
I20260808 15:49:09.093830 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.076540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.27.84.254:39485)
0808 15:49:09.078055 (+ 1515us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.078064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.078088 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:49:09.078171 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.078177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.078267 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.078405 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.079825 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.080651 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.082407 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.082894 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.083043 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.087087 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.087093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.087098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.087144 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.090877 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.091673 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.091677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.091680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.091814 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.092288 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.092292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.092294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.092621 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.092800 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.093352 (+ 552us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.093550 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1366,"thread_start_us":1235,"threads_started":1}
I20260808 15:49:09.093861 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.076464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:41386)
0808 15:49:09.077435 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.077505 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.077821 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.078494 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.078500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.079268 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.079636 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.079652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.080817 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.080825 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.082192 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.082206 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.083100 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.083107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.083290 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.083979 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.083997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.086874 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.091006 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.091016 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.091021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.091543 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.091933 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.091939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.091942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.092156 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.092766 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.092773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.092985 (+ 212us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.093244 (+ 259us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.093584 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":408,"threads_started":1}
I20260808 15:49:09.095548 6056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204148, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.253@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
I20260808 15:49:09.108952 6065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.096357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:59480)
0808 15:49:09.096519 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.096556 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.096658 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.096952 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.096955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.097435 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.097590 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.097599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.098643 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.098652 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.099527 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.099540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.100416 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.100421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.100580 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.101148 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.101167 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.103319 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.106384 (+ 3065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.106389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.106391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.106917 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.107378 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.107381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.107384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.107521 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.108192 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.108195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.108344 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.108450 (+ 106us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.108691 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:49:09.109320 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.096541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.27.84.253:41089)
0808 15:49:09.096695 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.096699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.096713 (+ 14us) server_negotiation.cc:407] Connection header received
0808 15:49:09.096751 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.096754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.096807 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.096915 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.097754 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.098462 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.099691 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.100194 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.100306 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.103496 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.103501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.103504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.103533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.106227 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.107077 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.107082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.107085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.107235 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.107671 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.107676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.107678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.108074 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.108211 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.108586 (+ 375us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.108899 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":57}
I20260808 15:49:09.114744 6056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Bootstrap starting.
I20260808 15:49:09.118053 6056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:09.119270 6056 log.cc:824] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:09.122294 6056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: No bootstrap required, opened a new log
I20260808 15:49:09.129923 6056 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } }
I20260808 15:49:09.130318 6056 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:09.130481 6056 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa99b51165794de98294ac7c51b86512, State: Initialized, Role: FOLLOWER
I20260808 15:49:09.131219 6056 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } }
I20260808 15:49:09.132294 6069 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } }
I20260808 15:49:09.132742 6069 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:49:09.133571 6056 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:49:09.143826 6088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:49:09.144117 6088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:49:09.146741 5952 master_runner.cc:428] Detected that this master fa99b51165794de98294ac7c51b86512 is joining an existing cluster
I20260808 15:49:09.146870 5952 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.84.253:41089
WARNING: no policy specified for oryx/127.27.84.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.252@KRBTEST.COM" created.
I20260808 15:49:09.187836 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.151373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.27.84.254:39485)
0808 15:49:09.151588 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.151597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.167782 (+ 16185us) server_negotiation.cc:407] Connection header received
0808 15:49:09.168200 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.168207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.168311 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.168382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.170555 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.171748 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.177630 (+ 5882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.178365 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.178486 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.181347 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.181354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.181359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.181407 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.184888 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.185655 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.185661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.185665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.185838 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.186248 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.186253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.186262 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.186553 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.186735 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.187198 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.187442 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:49:09.188037 6096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.151838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:41400)
0808 15:49:09.167657 (+ 15819us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.167677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.168042 (+ 365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.168573 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.168576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.169574 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.170335 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.170358 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.173087 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.173095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.175819 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.175834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.178844 (+ 3010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.178851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.179051 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.179728 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.179760 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.181039 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.185039 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.185043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.185047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.185522 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.185970 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.185975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.185979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.186127 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.186713 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.186718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.186924 (+ 206us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.187084 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.187431 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15682,"mutex_wait_us":1011,"spinlock_wait_cycles":91520,"thread_start_us":15477,"threads_started":1}
I20260808 15:49:09.197757 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.151366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.27.84.253:41089)
0808 15:49:09.151689 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.151698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.167100 (+ 15402us) server_negotiation.cc:407] Connection header received
0808 15:49:09.168156 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.168162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.168234 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.168303 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.170522 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.171741 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.173991 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.174877 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.175032 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.177677 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.177686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.177694 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.177737 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.182437 (+ 4700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.194108 (+ 11671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.194116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.194123 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.194321 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.194849 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.194857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.194861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.195327 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.195504 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.197000 (+ 1496us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.197260 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":72}
I20260808 15:49:09.198256 6095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.151736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:59492)
0808 15:49:09.166909 (+ 15173us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.166949 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.167997 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.168545 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.168551 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.169574 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.170357 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.170367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.172032 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.172039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.173823 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.173839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.175109 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.175117 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.175333 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.176092 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.176121 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.177381 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.182588 (+ 5207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.182596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.182602 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.183177 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.194460 (+ 11283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.194466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.194469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.194706 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.195576 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.195582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.195768 (+ 186us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.195930 (+ 162us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.196361 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14901,"mutex_wait_us":83,"spinlock_wait_cycles":1280,"thread_start_us":15971,"threads_started":1}
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.254@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.252.keytab.
Entry for principal oryx/127.27.84.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.252.keytab.
I20260808 15:49:09.223874 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:35957
--webserver_interface=127.27.84.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.252.keytab
--principal=oryx/127.27.84.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:49:09.225348 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.197671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:59494)
0808 15:49:09.203258 (+ 5587us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.203324 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.203526 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.204266 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.204275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.205270 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.205586 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.205602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.207223 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.207229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.208873 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.208890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.210136 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.210145 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.210371 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.211318 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.211347 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.216847 (+ 5500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.221251 (+ 4404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.221272 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.221282 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.222155 (+ 873us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.222816 (+ 661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.222822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.222826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.223044 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.223961 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.223971 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.224210 (+ 239us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.224465 (+ 255us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.224879 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5296,"thread_start_us":5127,"threads_started":1}
I20260808 15:49:09.226029 6063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.198065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.27.84.253:41089)
0808 15:49:09.200966 (+ 2901us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.200973 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.203736 (+ 2763us) server_negotiation.cc:407] Connection header received
0808 15:49:09.203852 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.203861 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.204040 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.204150 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.205769 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.207064 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.209085 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.209925 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.210089 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.217108 (+ 7019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.217113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.217117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.217178 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.221034 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.222401 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.222409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.222413 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.222656 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.223195 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.223201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.223207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.223773 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.223953 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.224613 (+ 660us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.224934 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2776}
I20260808 15:49:09.228312 5234 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.84.253:41089: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fa99b51165794de98294ac7c51b86512"
member_type: NON_VOTER
last_known_addr {
host: "127.27.84.253"
port: 41089
}
attrs {
promote: true
}
}
dest_uuid: "b173194b048f444294d077c949db5fc6"
cas_config_opid_index: -1
I20260808 15:49:09.230067 5234 consensus_queue.cc:237] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } }
I20260808 15:49:09.240073 6103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa99b51165794de98294ac7c51b86512 (127.27.84.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:09.241314 6013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:09.245893 6013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260808 15:49:09.245707 6103 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.84.253:41089
I20260808 15:49:09.247675 6109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:09.248693 5952 master.cc:537] Master@127.27.84.253:41089 shutting down...
W20260808 15:49:09.255191 5211 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260808 15:49:09.254936 6105 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:09.255431 6105 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:09.267558 5952 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260808 15:49:09.267638 5211 consensus_peers.cc:487] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 -> Peer fa99b51165794de98294ac7c51b86512 (127.27.84.253:41089): Couldn't send request to peer fa99b51165794de98294ac7c51b86512. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 15:49:09.268088 5952 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:09.268339 5952 tablet_replica.cc:330] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: stopping tablet replica
I20260808 15:49:09.275295 5952 master.cc:552] Master@127.27.84.253:41089 shutdown complete.
I20260808 15:49:09.275516 5952 master_runner.cc:305] Clearing existing system tablet
I20260808 15:49:09.276129 5952 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:09.276268 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:09.279211 5952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260808 15:49:09.281056 6115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:09.281635 5952 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:09.282052 5952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260808 15:49:09.282159 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
uuid: "fa99b51165794de98294ac7c51b86512"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "6a461f8b0c833df2da984d9ccf9a61a9"
server_key_iv: "fb0c58697b401fa408df2723db0cd3f8"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.286528 5952 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:09.292479 5952 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 15:49:09.292742 5952 log.cc:1196] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260808 15:49:09.293236 5952 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Deleting consensus metadata
I20260808 15:49:09.293569 5952 master_runner.cc:315] Copying system tablet from 127.27.84.254:39485
I20260808 15:49:09.296811 5952 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: tablet copy: Beginning tablet copy session from remote peer at address 127.27.84.254:39485
I20260808 15:49:09.312312 6066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.297984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.27.84.254:39485)
0808 15:49:09.298227 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.298237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.298840 (+ 603us) server_negotiation.cc:407] Connection header received
0808 15:49:09.299143 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.299149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.299231 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.299360 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.300365 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.301209 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.302815 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.303634 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.303773 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.306069 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.306085 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.306089 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.306125 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.309181 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.309933 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.309940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.309944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.310092 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.310568 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.310574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.310578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.310979 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.311161 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.311603 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.311857 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 15:49:09.312354 6122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.297945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:41414)
0808 15:49:09.298692 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.298724 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.298925 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.299397 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.299403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.299927 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.300172 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.300183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.301393 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.301399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.302529 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.302550 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.304049 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.304054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.304207 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.304941 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.304974 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.305830 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.309355 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.309361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.309366 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.309789 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.310225 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.310231 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.310239 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.310414 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.311123 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.311129 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.311325 (+ 196us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.311499 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.311865 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":447,"threads_started":1}
I20260808 15:49:09.313743 5265 tablet_copy_service.cc:140] P b173194b048f444294d077c949db5fc6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fa99b51165794de98294ac7c51b86512 ({username='oryx', principal='oryx/127.27.84.253@KRBTEST.COM'} at 127.0.0.1:41414)
I20260808 15:49:09.314309 5265 tablet_copy_service.cc:161] P b173194b048f444294d077c949db5fc6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fa99b51165794de98294ac7c51b86512 at {username='oryx', principal='oryx/127.27.84.253@KRBTEST.COM'} at 127.0.0.1:41414: session id = fa99b51165794de98294ac7c51b86512-00000000000000000000000000000000
I20260808 15:49:09.318192 5265 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 15:49:09.321208 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 15:49:09.327253 5952 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: tablet copy: Starting download of 0 data blocks...
I20260808 15:49:09.327684 5952 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: tablet copy: Starting download of 1 WAL segments...
I20260808 15:49:09.331346 5952 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 15:49:09.335328 5265 tablet_copy_service.cc:342] P b173194b048f444294d077c949db5fc6: Request end of tablet copy session fa99b51165794de98294ac7c51b86512-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.84.253@KRBTEST.COM'} at 127.0.0.1:41414
I20260808 15:49:09.335659 5265 tablet_copy_service.cc:433] P b173194b048f444294d077c949db5fc6: ending tablet copy session fa99b51165794de98294ac7c51b86512-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fa99b51165794de98294ac7c51b86512
I20260808 15:49:09.349366 5952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:09.356544 6123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:09.367151 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:09.372785 5952 server_base.cc:1034] running on GCE node
W20260808 15:49:09.373031 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:09.373587 5952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:09.374349 5952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:09.375710 5952 hybrid_clock.cc:648] HybridClock initialized: now 1786204149375646 us; error 40 us; skew 500 ppm
I20260808 15:49:09.375963 5952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204149, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:09.378262 5952 init.cc:377] Logged in from keytab as oryx/127.27.84.253@KRBTEST.COM (short username oryx)
I20260808 15:49:09.380192 5952 webserver.cc:460] Webserver started at http://127.27.84.253:38309/ using document root <none> and password file <none>
I20260808 15:49:09.380630 5952 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:09.380730 5952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:09.384261 5952 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260808 15:49:09.390527 6133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:09.391201 5952 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260808 15:49:09.391424 5952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
uuid: "fa99b51165794de98294ac7c51b86512"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "6a461f8b0c833df2da984d9ccf9a61a9"
server_key_iv: "fb0c58697b401fa408df2723db0cd3f8"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.391650 5952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:09.418937 5952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:09.423316 5952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:09.450045 5952 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.253:41089
I20260808 15:49:09.450101 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.253:41089 every 8 connection(s)
I20260808 15:49:09.451167 5952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
I20260808 15:49:09.456552 6212 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:49:09.460927 6212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Bootstrap starting.
I20260808 15:49:09.478461 6212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:49:09.479094 6212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: Bootstrap complete.
I20260808 15:49:09.480311 6212 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } }
I20260808 15:49:09.480587 6212 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fa99b51165794de98294ac7c51b86512, State: Initialized, Role: LEARNER
I20260808 15:49:09.480841 6212 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } }
I20260808 15:49:09.482021 6212 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:09.481779 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:09.482323 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:49:09.500058 6231 catalog_manager.cc:1259] Loaded cluster ID: cf9b92ee36964699ba0615c5f9a7497b
I20260808 15:49:09.500268 6231 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: loading cluster ID for follower catalog manager: success
I20260808 15:49:09.504033 6231 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: acquiring CA information for follower catalog manager: success
I20260808 15:49:09.506865 6231 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:49:09.521169 5945 heartbeater.cc:503] Master 127.27.84.254:39485 was elected leader, sending a full tablet report...
W20260808 15:49:09.689710 6107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:09.690084 6107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:09.690183 6107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:09.698546 6107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:09.698735 6107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:09.698800 6107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:09.698860 6107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:09.698905 6107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:09.709851 6107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.252.keytab
--principal=oryx/127.27.84.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.84.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:09.711406 6107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:09.713013 6107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:09.726336 6238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:09.726431 6241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:09.726362 6237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:09.728199 6107 server_base.cc:1034] running on GCE node
I20260808 15:49:09.729027 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:09.730553 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:09.731848 6107 hybrid_clock.cc:648] HybridClock initialized: now 1786204149731800 us; error 41 us; skew 500 ppm
I20260808 15:49:09.732223 6107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204149, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:09.735992 6107 init.cc:377] Logged in from keytab as oryx/127.27.84.252@KRBTEST.COM (short username oryx)
I20260808 15:49:09.737783 6107 webserver.cc:460] Webserver started at http://127.27.84.252:34379/ using document root <none> and password file <none>
I20260808 15:49:09.738523 6107 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:09.738755 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:09.739130 6107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:09.742255 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/instance:
uuid: "f83ced808e574044a6d122cfe7d700ab"
format_stamp: "Formatted at 2026-08-08 15:49:09 on dist-test-slave-twwt"
server_key: "5875506668144d493b970a3ce4ae3251"
server_key_iv: "1d5c56128f51a5705748076bc913b338"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.743109 6107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal/instance:
uuid: "f83ced808e574044a6d122cfe7d700ab"
format_stamp: "Formatted at 2026-08-08 15:49:09 on dist-test-slave-twwt"
server_key: "5875506668144d493b970a3ce4ae3251"
server_key_iv: "1d5c56128f51a5705748076bc913b338"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.749221 6107 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 15:49:09.753298 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:09.755055 6107 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260808 15:49:09.755249 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
uuid: "f83ced808e574044a6d122cfe7d700ab"
format_stamp: "Formatted at 2026-08-08 15:49:09 on dist-test-slave-twwt"
server_key: "5875506668144d493b970a3ce4ae3251"
server_key_iv: "1d5c56128f51a5705748076bc913b338"
server_key_version: "encryptionkey@0"
I20260808 15:49:09.755539 6107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:09.774730 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.768127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:59510)
0808 15:49:09.768583 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.768609 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.768778 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.769341 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.769346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.769378 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:09.769583 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.769593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.771161 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.771168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.772426 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.772441 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.773778 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.773784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.773974 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.774062 (+ 88us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.774177 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.774337 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":247,"threads_started":1}
I20260808 15:49:09.774746 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.768300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59510 (local address 127.27.84.253:41089)
0808 15:49:09.768991 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.768996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.769013 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:49:09.769078 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.769082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.769154 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.769296 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:09.769740 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.771000 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.772628 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.773570 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.773738 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.774001 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.774212 (+ 211us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.774377 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":386,"threads_started":1}
I20260808 15:49:09.775938 6179 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260808 15:49:09.776655 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b173194b048f444294d077c949db5fc6. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:09.777091 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:49:09.782414 6107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:09.787482 6107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:09.787738 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:09.811389 6107 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.252:35957
I20260808 15:49:09.811432 6327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.252:35957 every 8 connection(s)
I20260808 15:49:09.813021 6107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
I20260808 15:49:09.814194 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 6107
I20260808 15:49:09.818182 6328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:09.829030 6328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.252@KRBTEST.COM
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204149, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.252@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:09.844856 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.817054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.27.84.252:35957)
0808 15:49:09.819200 (+ 2146us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.819217 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.819716 (+ 499us) server_negotiation.cc:407] Connection header received
0808 15:49:09.822033 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.822064 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.822689 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.823212 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.824829 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.827058 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.828761 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.830077 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.830289 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.835929 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.835937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.835948 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.835985 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.839949 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.840934 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.840945 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.840956 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.841179 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.841600 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.841606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.841610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.842434 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.842780 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.843243 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.843761 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":1492,"threads_started":1}
I20260808 15:49:09.850548 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.832724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.27.84.254:39485)
0808 15:49:09.833356 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.833365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.833951 (+ 586us) server_negotiation.cc:407] Connection header received
0808 15:49:09.834231 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.834236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.834304 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.834497 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.835899 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.836982 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.839460 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.839925 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.840089 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.844224 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.844229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.844232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.844265 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.847360 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.848328 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.848332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.848335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.848466 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.848874 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.848878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.848880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.849155 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.849325 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.849793 (+ 468us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.850029 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":360,"threads_started":1}
I20260808 15:49:09.850596 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.832718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:41424)
0808 15:49:09.833810 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.833849 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.834078 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.834502 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.834509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.835351 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.835736 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.835750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.837146 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.837152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.839281 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.839298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.840132 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.840140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.840284 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.841085 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.841108 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.844023 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.847504 (+ 3481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.847518 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.847525 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.848149 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.848585 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.848590 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.848594 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.848741 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.849288 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.849296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.849483 (+ 187us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.849682 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.850029 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"spinlock_wait_cycles":23424,"thread_start_us":604,"threads_started":1}
I20260808 15:49:09.852131 6328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204149, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.252@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
I20260808 15:49:09.865103 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.852844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:59520)
0808 15:49:09.853022 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.853044 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.853132 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.853427 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.853431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.853745 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.853907 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.853915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.855141 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.855145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.856101 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.856112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.856823 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.856827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.856968 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.857482 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.857499 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.859693 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.862842 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.862847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.862850 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.863287 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.863631 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.863634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.863636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.863778 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.864290 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.864295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.864446 (+ 151us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.864569 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.864816 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:49:09.865118 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.852976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59520 (local address 127.27.84.253:41089)
0808 15:49:09.853161 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.853166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.853197 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:49:09.853247 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.853251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.853314 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.853391 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.854058 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.855002 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.856234 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.856648 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.856741 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.859902 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.859907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.859910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.859941 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.862694 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.863391 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.863395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.863397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.863526 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.863873 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.863877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.863879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.864140 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.864252 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.864655 (+ 403us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.864861 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:49:09.866139 6328 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204149, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.252@KRBTEST.COM for oryx/127.27.84.252@KRBTEST.COM
I20260808 15:49:09.879581 6334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.866842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.0.0.1:53910)
0808 15:49:09.867027 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.867047 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.867171 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.867491 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.867495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.867921 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.868101 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.868110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.869058 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.869062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.870001 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.870014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.870798 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.870803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.870944 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.871501 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.871519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.873589 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.877019 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.877023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.877026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.877594 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.878025 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.878030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.878032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.878225 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.878788 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.878792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.878920 (+ 128us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.879049 (+ 129us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.879265 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 15:49:09.879820 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.866917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.27.84.252:35957)
0808 15:49:09.867050 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.867056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.867105 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:49:09.867279 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.867284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.867358 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.867457 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.868228 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.868922 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.870130 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.870594 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.870748 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.873760 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.873764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.873768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.873804 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.876869 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.877718 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.877722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.877724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.877893 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.878330 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.878333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.878335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.878649 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.878774 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.879191 (+ 417us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.879399 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":48}
I20260808 15:49:09.885126 6328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Bootstrap starting.
I20260808 15:49:09.888954 6328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:09.890296 6328 log.cc:824] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:09.893491 6328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: No bootstrap required, opened a new log
I20260808 15:49:09.902053 6328 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } }
I20260808 15:49:09.902469 6328 raft_consensus.cc:374] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:09.902712 6328 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83ced808e574044a6d122cfe7d700ab, State: Initialized, Role: FOLLOWER
I20260808 15:49:09.903419 6328 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } }
I20260808 15:49:09.904500 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } } }
I20260808 15:49:09.904903 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:49:09.905612 6328 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: configured and running, proceeding with master startup.
W20260808 15:49:09.915932 6353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:49:09.916127 6353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:49:09.920639 6107 master_runner.cc:428] Detected that this master f83ced808e574044a6d122cfe7d700ab is joining an existing cluster
I20260808 15:49:09.920796 6107 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.84.252:35957
I20260808 15:49:09.945114 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.926437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.27.84.254:39485)
0808 15:49:09.926662 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.926670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.926854 (+ 184us) server_negotiation.cc:407] Connection header received
0808 15:49:09.927211 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.927217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.927341 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.927421 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.929719 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.930997 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.933353 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.934183 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.934296 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.937481 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.937488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.937493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.937536 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.941469 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.942542 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.942548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.942551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.942759 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.943254 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.943259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.943263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.943591 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.943754 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.944445 (+ 691us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.944721 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 15:49:09.945417 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.925844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.27.84.253:41089)
0808 15:49:09.926043 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.926050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.927364 (+ 1314us) server_negotiation.cc:407] Connection header received
0808 15:49:09.927669 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.927675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.927754 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.927847 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.930367 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.931484 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.933871 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.934677 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.934782 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.937997 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.938004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.938009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.938052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.941740 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.942847 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.942856 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.942861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.943029 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.943537 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.943543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.943547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.943928 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.944053 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.944701 (+ 648us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.944967 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:49:09.947104 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.925780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:59526)
0808 15:49:09.927268 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.927285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.927516 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.928091 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.928095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.929092 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.930167 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.930188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.931686 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.931691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.933666 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.933688 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.935151 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.935158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.935333 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.936205 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.936233 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.937702 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.941929 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.941934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.941938 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.942672 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.943185 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.943190 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.943194 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.943397 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.944150 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.944155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.944384 (+ 229us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.945648 (+ 1264us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.946035 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":486,"threads_started":1}
I20260808 15:49:09.947762 6367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.925687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:41434)
0808 15:49:09.926707 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.926734 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.927008 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.927657 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.927661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.929229 (+ 1568us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.929563 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.929573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.931237 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.931243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.933183 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.933198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.934648 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.934656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.934853 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.935681 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.935706 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.937263 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.941669 (+ 4406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.941678 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.941683 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.942397 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.942943 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.942950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.942955 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.943131 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.943853 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.943858 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.944093 (+ 235us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.945878 (+ 1785us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.946396 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":1217,"threads_started":1}
I20260808 15:49:09.953321 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.927690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.27.84.252:35957)
0808 15:49:09.927878 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.927884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.930258 (+ 2374us) server_negotiation.cc:407] Connection header received
0808 15:49:09.930465 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.930470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.930530 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.930587 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.932307 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.933392 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.936028 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.936854 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.936952 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.940267 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.940275 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.940280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.940315 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.949554 (+ 9239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.950413 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.950419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.950424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.950593 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.951102 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.951109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.951113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.951531 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.951718 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.952369 (+ 651us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.952764 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":33}
I20260808 15:49:09.953358 6371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.927337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.0.0.1:53922)
0808 15:49:09.930162 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.930180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.930354 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.930792 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.930796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.931695 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.932139 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.932156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.933693 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.933699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.935722 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.935739 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.937312 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.937321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.937511 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.938539 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.938562 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.940067 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.949696 (+ 9629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.949701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.949714 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.950277 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.950805 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.950811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.950818 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.950982 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.951821 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.951826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.952024 (+ 198us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.952190 (+ 166us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.952577 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"thread_start_us":1820,"threads_started":1}
I20260808 15:49:09.956337 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 5328
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.254@KRBTEST.COM for oryx/127.27.84.252@KRBTEST.COM
Aug 08 15:49:09 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.193@KRBTEST.COM
I20260808 15:49:09.977881 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.961900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.0.0.1:53938)
0808 15:49:09.962196 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:09.962230 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:09.962344 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:09.962894 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:09.962901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:09.963402 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:09.963578 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.963589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.965157 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.965163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.966540 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:09.966552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.967569 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.967575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:09.967758 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.968339 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:09.968357 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:09.970514 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:09.974895 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.974901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.974904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.975380 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.975813 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:09.975819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:09.975823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:09.975980 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:09.976671 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:09.976676 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:09.976899 (+ 223us) client_negotiation.cc:769] Sending connection context
0808 15:49:09.977056 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 15:49:09.977381 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260808 15:49:09.977907 6331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.962111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.27.84.252:35957)
0808 15:49:09.962298 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.962304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.962324 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:49:09.962509 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.962517 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.962716 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.962838 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.963790 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.965038 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.966751 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.967378 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.967530 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.970854 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.970861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.970866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.970914 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.974716 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.975533 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.975537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.975539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.975685 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.976107 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.976112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.976116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.976526 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.976707 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.977156 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.977399 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:49:09.979619 5234 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.84.252:35957: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f83ced808e574044a6d122cfe7d700ab"
member_type: NON_VOTER
last_known_addr {
host: "127.27.84.252"
port: 35957
}
attrs {
promote: true
}
}
dest_uuid: "b173194b048f444294d077c949db5fc6"
cas_config_opid_index: 5
I20260808 15:49:09.980865 5234 consensus_queue.cc:237] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } }
I20260808 15:49:09.982262 6372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:09.957508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.27.84.193:33515)
0808 15:49:09.959109 (+ 1601us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:09.959119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:09.959151 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:49:09.959229 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:09.959242 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:09.959621 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:09.959810 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:09.961646 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.963767 (+ 2121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.965087 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:09.966038 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:09.966174 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:09.972580 (+ 6406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:09.972599 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:09.972622 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:09.972676 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:09.977435 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.979105 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.979115 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.979118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.979307 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:09.979929 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:09.979934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:09.979938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:09.980466 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:09.980895 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:09.981242 (+ 347us) server_negotiation.cc:299] Negotiation successful
0808 15:49:09.981561 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1394,"thread_start_us":1219,"threads_started":1}
I20260808 15:49:09.982653 6336 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:09.982749 6255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f83ced808e574044a6d122cfe7d700ab (127.27.84.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:09.983028 6336 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } }
I20260808 15:49:09.983932 5530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:09.985018 6336 leader_election.cc:288] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b173194b048f444294d077c949db5fc6 (127.27.84.254:39485), fa99b51165794de98294ac7c51b86512 (127.27.84.253:41089)
I20260808 15:49:09.986104 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f83ced808e574044a6d122cfe7d700ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa99b51165794de98294ac7c51b86512" is_pre_election: true
I20260808 15:49:09.986162 5247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f83ced808e574044a6d122cfe7d700ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b173194b048f444294d077c949db5fc6" is_pre_election: true
I20260808 15:49:09.987736 6252 leader_election.cc:302] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f83ced808e574044a6d122cfe7d700ab; no voters: b173194b048f444294d077c949db5fc6, fa99b51165794de98294ac7c51b86512
I20260808 15:49:09.988404 6295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:09.989142 6179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 LEARNER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 15:49:09.990899 6374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260808 15:49:09.992822 6295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260808 15:49:09.993037 6336 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260808 15:49:09.994558 6179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f83ced808e574044a6d122cfe7d700ab (127.27.84.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:09.997205 6374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:09.998709 6255 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.84.252:35957
I20260808 15:49:09.999622 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.000128 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.002075 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.002420 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: LEARNER
I20260808 15:49:10.004181 6374 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: attempting to promote NON_VOTER fa99b51165794de98294ac7c51b86512 to VOTER
I20260808 15:49:10.005190 6374 consensus_queue.cc:237] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } }
I20260808 15:49:10.007783 6179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 LEARNER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 15:49:10.007846 6294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260808 15:49:10.008584 6377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260808 15:49:10.010257 6377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:10.012183 6374 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fa99b51165794de98294ac7c51b86512 (127.27.84.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.013671 6179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fa99b51165794de98294ac7c51b86512 (127.27.84.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.017336 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.017696 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.018148 6215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.018457 6215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:49:10.019230 6374 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.019511 6374 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.023041 6107 master.cc:537] Master@127.27.84.252:35957 shutting down...
I20260808 15:49:10.037236 6294 raft_consensus.cc:858] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } }
W20260808 15:49:10.037521 6294 raft_consensus.cc:1565] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } }
W20260808 15:49:10.040248 6259 tablet.cc:2369] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:10.049644 6294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fa99b51165794de98294ac7c51b86512 (127.27.84.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:10.054972 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b173194b048f444294d077c949db5fc6. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:10.055261 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 15:49:10.055486 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:10.055684 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 15:49:10.055783 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.056061 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.056005 6385 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } }
I20260808 15:49:10.056420 6385 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260808 15:49:10.056633 6295 raft_consensus.cc:1467] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
I20260808 15:49:10.056843 6295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 NON_PARTICIPANT]: Deduplicated request from leader. Original: 0.0->[1.1-1.6] Dedup: 1.5->[1.6-1.6]
I20260808 15:49:10.057395 6295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f83ced808e574044a6d122cfe7d700ab (127.27.84.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.062122 6385 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: NON_VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: true } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.062435 6385 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: LEARNER
W20260808 15:49:10.065362 5209 consensus_peers.cc:487] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 -> Peer f83ced808e574044a6d122cfe7d700ab (127.27.84.252:35957): Couldn't send request to peer f83ced808e574044a6d122cfe7d700ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260808 15:49:10.066469 6107 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Raft consensus shutting down.
I20260808 15:49:10.067075 6107 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Raft consensus is shut down!
I20260808 15:49:10.067266 6107 tablet_replica.cc:330] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: stopping tablet replica
I20260808 15:49:10.086311 6107 master.cc:552] Master@127.27.84.252:35957 shutdown complete.
I20260808 15:49:10.086421 6107 master_runner.cc:305] Clearing existing system tablet
I20260808 15:49:10.086846 6107 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:10.086956 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:10.090137 6107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260808 15:49:10.092062 6393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:10.092605 6107 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:10.092936 6107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260808 15:49:10.093053 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
uuid: "f83ced808e574044a6d122cfe7d700ab"
format_stamp: "Formatted at 2026-08-08 15:49:09 on dist-test-slave-twwt"
server_key: "5875506668144d493b970a3ce4ae3251"
server_key_iv: "1d5c56128f51a5705748076bc913b338"
server_key_version: "encryptionkey@0"
I20260808 15:49:10.098282 6107 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:10.105501 6107 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260808 15:49:10.105674 6107 log.cc:1196] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260808 15:49:10.106241 6107 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Deleting consensus metadata
I20260808 15:49:10.106487 6107 master_runner.cc:315] Copying system tablet from 127.27.84.254:39485
I20260808 15:49:10.115617 6107 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: tablet copy: Beginning tablet copy session from remote peer at address 127.27.84.254:39485
I20260808 15:49:10.124370 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 5328
I20260808 15:49:10.137579 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.116588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.27.84.254:39485)
0808 15:49:10.116821 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:10.116826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:10.121009 (+ 4183us) server_negotiation.cc:407] Connection header received
0808 15:49:10.121273 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:10.121279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:10.121373 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:10.121511 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:10.123020 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.124277 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.127102 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.127769 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.127911 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.131509 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:10.131514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:10.131517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:10.131551 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:10.134811 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.135475 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.135481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.135485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.135629 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.136034 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.136039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.136043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.136375 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:10.136540 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:10.136978 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 15:49:10.137186 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260808 15:49:10.137609 6402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.116476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:41446)
0808 15:49:10.120853 (+ 4377us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:10.120876 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:10.121098 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:10.121682 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:10.121691 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:10.122357 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:10.122827 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.122847 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.124448 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.124456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.126543 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.126562 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.127957 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.127966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.128220 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.129110 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:10.129141 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:10.130591 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:10.134944 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.134950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.134954 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.135342 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.135755 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.135762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.135769 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.135911 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.136501 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:10.136506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:10.136685 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 15:49:10.136848 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 15:49:10.137161 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4197,"thread_start_us":4076,"threads_started":1}
I20260808 15:49:10.138681 5265 tablet_copy_service.cc:140] P b173194b048f444294d077c949db5fc6: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f83ced808e574044a6d122cfe7d700ab ({username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:41446)
I20260808 15:49:10.138873 5265 tablet_copy_service.cc:161] P b173194b048f444294d077c949db5fc6: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f83ced808e574044a6d122cfe7d700ab at {username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:41446: session id = f83ced808e574044a6d122cfe7d700ab-00000000000000000000000000000000
I20260808 15:49:10.141325 5265 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Tablet Copy: opened 0 blocks and 1 log segments
I20260808 15:49:10.143174 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260808 15:49:10.148150 6107 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: tablet copy: Starting download of 0 data blocks...
I20260808 15:49:10.148465 6107 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: tablet copy: Starting download of 1 WAL segments...
I20260808 15:49:10.151940 6107 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260808 15:49:10.155486 5265 tablet_copy_service.cc:342] P b173194b048f444294d077c949db5fc6: Request end of tablet copy session f83ced808e574044a6d122cfe7d700ab-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:41446
I20260808 15:49:10.155637 5265 tablet_copy_service.cc:433] P b173194b048f444294d077c949db5fc6: ending tablet copy session f83ced808e574044a6d122cfe7d700ab-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f83ced808e574044a6d122cfe7d700ab
I20260808 15:49:10.158452 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:33515
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:39485
--webserver_port=40179
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:49:10.168919 6107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:10.171957 6404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:10.177573 6406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:10.185182 6107 server_base.cc:1034] running on GCE node
W20260808 15:49:10.186904 6408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:10.187419 6107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:10.188172 6107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:10.189383 6107 hybrid_clock.cc:648] HybridClock initialized: now 1786204150189337 us; error 36 us; skew 500 ppm
I20260808 15:49:10.189685 6107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:10 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204150, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:10.191874 6107 init.cc:377] Logged in from keytab as oryx/127.27.84.252@KRBTEST.COM (short username oryx)
I20260808 15:49:10.195706 6107 webserver.cc:460] Webserver started at http://127.27.84.252:36631/ using document root <none> and password file <none>
I20260808 15:49:10.196051 6107 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:10.196166 6107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:10.199085 6107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260808 15:49:10.200932 6422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:10.201392 6107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260808 15:49:10.201531 6107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
uuid: "f83ced808e574044a6d122cfe7d700ab"
format_stamp: "Formatted at 2026-08-08 15:49:09 on dist-test-slave-twwt"
server_key: "5875506668144d493b970a3ce4ae3251"
server_key_iv: "1d5c56128f51a5705748076bc913b338"
server_key_version: "encryptionkey@0"
I20260808 15:49:10.201725 6107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:10.219290 6107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:10.223371 6107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:10.249714 6107 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.252:35957
I20260808 15:49:10.249735 6501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.252:35957 every 8 connection(s)
I20260808 15:49:10.250864 6107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
I20260808 15:49:10.258028 6503 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:49:10.260629 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Bootstrap starting.
I20260808 15:49:10.272811 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:49:10.273339 6503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Bootstrap complete.
I20260808 15:49:10.274262 6503 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } }
I20260808 15:49:10.274523 6503 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f83ced808e574044a6d122cfe7d700ab, State: Initialized, Role: LEARNER
I20260808 15:49:10.274791 6503 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } }
I20260808 15:49:10.275700 6514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.276065 6503 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:10.276119 6514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: LEARNER
I20260808 15:49:10.291894 6528 catalog_manager.cc:1259] Loaded cluster ID: cf9b92ee36964699ba0615c5f9a7497b
I20260808 15:49:10.292037 6528 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: loading cluster ID for follower catalog manager: success
I20260808 15:49:10.294864 6528 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: acquiring CA information for follower catalog manager: success
I20260808 15:49:10.296790 6528 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:49:10.505056 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.505421 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.513420 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.505508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.0.0.1:53954)
0808 15:49:10.506171 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:10.506199 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:10.506368 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:10.506962 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:10.506966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:10.506994 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:10.507354 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.507368 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.509321 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.509327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.510736 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.510752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.512731 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.512736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.512890 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.512930 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:49:10.513015 (+ 85us) client_negotiation.cc:240] Negotiation successful
0808 15:49:10.513152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":328,"threads_started":1}
I20260808 15:49:10.513693 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.505687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53954 (local address 127.27.84.252:35957)
0808 15:49:10.506517 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:10.506523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:10.506546 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:49:10.506648 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:10.506653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:10.506765 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:10.506912 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:10.507544 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.509153 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.510953 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.512496 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.512642 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.512941 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:10.513069 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:49:10.513240 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":477,"threads_started":1}
I20260808 15:49:10.514739 6462 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260808 15:49:10.515360 6514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b173194b048f444294d077c949db5fc6. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: NON_VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: true } } }
I20260808 15:49:10.515995 6514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: LEARNER
I20260808 15:49:10.516384 6376 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: attempting to promote NON_VOTER f83ced808e574044a6d122cfe7d700ab to VOTER
I20260808 15:49:10.517261 6376 consensus_queue.cc:237] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } }
I20260808 15:49:10.519686 6179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 15:49:10.519730 6462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 LEARNER]: Refusing update from remote peer b173194b048f444294d077c949db5fc6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260808 15:49:10.520396 6376 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 15:49:10.520694 6255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260808 15:49:10.523265 6255 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f83ced808e574044a6d122cfe7d700ab (127.27.84.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.525022 6462 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f83ced808e574044a6d122cfe7d700ab (127.27.84.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.527097 6179 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f83ced808e574044a6d122cfe7d700ab (127.27.84.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.529810 6514 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.530135 6514 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:49:10.530047 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.530349 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.531090 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.531479 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.531744 6376 sys_catalog.cc:455] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.532087 6376 sys_catalog.cc:458] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:10.532667 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b173194b048f444294d077c949db5fc6" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:10.533013 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: FOLLOWER
W20260808 15:49:10.610113 6403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:10.610497 6403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:10.610721 6403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:10.618532 6403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:10.618744 6403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:10.618938 6403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:10.629535 6403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:33515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=40179
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:10.631175 6403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:10.632751 6403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:10.654757 6544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:10.657069 6545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:10.657263 6548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:10.658931 6403 server_base.cc:1034] running on GCE node
I20260808 15:49:10.659713 6403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:10.660893 6403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:10.662164 6403 hybrid_clock.cc:648] HybridClock initialized: now 1786204150662089 us; error 67 us; skew 500 ppm
I20260808 15:49:10.662563 6403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:10 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204150, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:10.666241 6403 init.cc:377] Logged in from keytab as oryx/127.27.84.193@KRBTEST.COM (short username oryx)
I20260808 15:49:10.668195 6403 webserver.cc:460] Webserver started at http://127.27.84.193:40179/ using document root <none> and password file <none>
I20260808 15:49:10.668993 6403 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:10.669171 6403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:10.675642 6403 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 15:49:10.679375 6557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:10.681041 6403 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 15:49:10.681223 6403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "ee5ec6fb7ca846df819e9a930eff7e5f"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "9f0f71ab679cdcb2d980993146b306f0"
server_key_iv: "85a4ed26662a2dd13d1a16e3186c5f94"
server_key_version: "encryptionkey@0"
I20260808 15:49:10.682435 6403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:10.707772 6403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:10.713099 6403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:10.713439 6403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:10.714720 6403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:10.716451 6403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:10.716549 6403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:10.716715 6403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:10.716837 6403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:10.780534 6403 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:33515
I20260808 15:49:10.780583 6713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:33515 every 8 connection(s)
I20260808 15:49:10.782157 6403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:10.788264 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 6403
I20260808 15:49:10.788669 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 5544
Aug 08 15:49:10 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.194@KRBTEST.COM
Aug 08 15:49:10 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204150, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
Aug 08 15:49:10 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204150, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
Aug 08 15:49:10 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204150, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.252@KRBTEST.COM
I20260808 15:49:10.848866 6726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.790353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.27.84.194:40679)
0808 15:49:10.820027 (+ 29674us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:10.820037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:10.820069 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:49:10.820146 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:10.820157 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:10.820463 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:10.820640 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:10.823999 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.825782 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.827828 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.829021 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.829149 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.837399 (+ 8250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:10.837422 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:10.837442 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:10.837517 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:10.843086 (+ 5569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.844348 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.844360 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.844366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.844562 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.845057 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.845063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.845068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.845564 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:10.845902 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:10.847212 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0808 15:49:10.847527 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29476,"thread_start_us":29248,"threads_started":1}
I20260808 15:49:10.850832 5728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:10.856098 6721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.786143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:33307 (local address 127.27.84.254:39485)
0808 15:49:10.816764 (+ 30621us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:10.816769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:10.816792 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:49:10.817521 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:10.817526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:10.817633 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:10.817778 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:10.820206 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.821453 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.826347 (+ 4894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.827056 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.827158 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.840362 (+ 13204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:10.840370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:10.840375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:10.840423 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:10.844721 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.847189 (+ 2468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.847196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.847199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.847391 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.848015 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.848024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.848029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.848483 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:10.848653 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:10.855151 (+ 6498us) server_negotiation.cc:299] Negotiation successful
0808 15:49:10.855516 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30496,"thread_start_us":30624,"threads_started":1}
I20260808 15:49:10.858536 6532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.824084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:54915 (local address 127.27.84.252:35957)
0808 15:49:10.824335 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:10.824343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:10.835457 (+ 11114us) server_negotiation.cc:407] Connection header received
0808 15:49:10.835662 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:10.835678 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:10.835782 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:10.835891 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:10.837765 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.839130 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.842920 (+ 3790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.843984 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.844113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.849770 (+ 5657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:10.849780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:10.849787 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:10.849842 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:10.854271 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.855368 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.855375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.855378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.855558 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.856089 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.856097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.856103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.856825 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:10.856983 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:10.857683 (+ 700us) server_negotiation.cc:299] Negotiation successful
0808 15:49:10.857984 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:49:10.860474 6727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.786918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:33307)
0808 15:49:10.810914 (+ 23996us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:10.810979 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:10.817177 (+ 6198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:10.818106 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:10.818116 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:10.818917 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:10.819962 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.819996 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.821689 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.821696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.826135 (+ 4439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.826151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.827515 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.827524 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.827896 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.833635 (+ 5739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:10.833668 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:10.838314 (+ 4646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:10.844922 (+ 6608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.844931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.844956 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.847030 (+ 2074us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.847570 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.847575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.847579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.847840 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.853468 (+ 5628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:10.853481 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:10.854085 (+ 604us) client_negotiation.cc:769] Sending connection context
0808 15:49:10.857637 (+ 3552us) client_negotiation.cc:240] Negotiation successful
0808 15:49:10.858478 (+ 841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23586,"thread_start_us":36197,"threads_started":1}
I20260808 15:49:10.861284 6730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.824525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.27.84.193:54915)
0808 15:49:10.835309 (+ 10784us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:10.835354 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:10.835510 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:10.836132 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:10.836140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:10.836853 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:10.837464 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.837483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.839370 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.839378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.842681 (+ 3303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.842704 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.844509 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.844518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.844759 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.845440 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:10.845475 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:10.849494 (+ 4019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:10.854456 (+ 4962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.854463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.854470 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.855174 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.855741 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.855749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.855752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.855945 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.857079 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:10.857086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:10.857357 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 15:49:10.859986 (+ 2629us) client_negotiation.cc:240] Negotiation successful
0808 15:49:10.860381 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10619,"thread_start_us":10430,"threads_started":1}
I20260808 15:49:10.862663 6715 heartbeater.cc:347] Connected to a master server at 127.27.84.252:35957
I20260808 15:49:10.863030 6715 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:10.864120 6715 heartbeater.cc:511] Master 127.27.84.252:35957 requested a full tablet report, sending...
I20260808 15:49:10.864884 6728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.791825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:39073 (local address 127.27.84.253:41089)
0808 15:49:10.815251 (+ 23426us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:10.815262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:10.823738 (+ 8476us) server_negotiation.cc:407] Connection header received
0808 15:49:10.823926 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:10.823934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:10.824025 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:10.824141 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:10.828628 (+ 4487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.829979 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.832315 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.833132 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.833270 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.848148 (+ 14878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:10.848156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:10.848161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:10.848207 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:10.853243 (+ 5036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.861681 (+ 8438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.861692 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.861698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.861927 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:10.862510 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:10.862518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:10.862524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:10.863025 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:10.863204 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:10.863910 (+ 706us) server_negotiation.cc:299] Negotiation successful
0808 15:49:10.864268 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23193,"thread_start_us":23463,"threads_started":1}
I20260808 15:49:10.865149 6717 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39485
I20260808 15:49:10.865345 6717 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:10.865897 6717 heartbeater.cc:511] Master 127.27.84.254:39485 requested a full tablet report, sending...
I20260808 15:49:10.867101 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:10.791131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.27.84.193:39073)
0808 15:49:10.823619 (+ 32488us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:10.823639 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:10.823790 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:10.824486 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:10.824490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:10.825133 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:10.825536 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.825549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.830202 (+ 4653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.830208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.832043 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:10.832062 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:10.835651 (+ 3589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:10.835659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:10.835877 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:10.838637 (+ 2760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:10.838664 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:10.846383 (+ 7719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:10.854556 (+ 8173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.854563 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.854570 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.857456 (+ 2886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.862126 (+ 4670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:10.862134 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:10.862140 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:10.862350 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:10.863323 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:10.863330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:10.863554 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 15:49:10.865095 (+ 1541us) client_negotiation.cc:240] Negotiation successful
0808 15:49:10.865550 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32321,"thread_start_us":19136,"threads_started":1}
I20260808 15:49:10.868022 5233 ts_manager.cc:194] Re-registered known tserver with Master: ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515)
I20260808 15:49:10.868438 6720 heartbeater.cc:347] Connected to a master server at 127.27.84.253:41089
I20260808 15:49:10.868566 6720 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:10.869002 6720 heartbeater.cc:511] Master 127.27.84.253:41089 requested a full tablet report, sending...
I20260808 15:49:10.869645 5233 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:33307
I20260808 15:49:10.871325 6156 ts_manager.cc:194] Registered new tserver with Master: ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515)
I20260808 15:49:10.872349 6449 ts_manager.cc:194] Registered new tserver with Master: ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515)
I20260808 15:49:10.959923 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 5544
I20260808 15:49:10.990516 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:40679
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:39485
--webserver_port=42325
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:11.354301 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:11.354660 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:11.354887 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:11.362403 6732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:11.362573 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:11.362802 6732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:11.372913 6732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:40679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=42325
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:11.374363 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:11.375936 6732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:11.391425 6742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:11.391527 6743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:11.391492 6745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:11.392350 6732 server_base.cc:1034] running on GCE node
I20260808 15:49:11.393148 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:11.394317 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:11.395599 6732 hybrid_clock.cc:648] HybridClock initialized: now 1786204151395505 us; error 106 us; skew 500 ppm
I20260808 15:49:11.396023 6732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:11 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204151, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:11.399871 6732 init.cc:377] Logged in from keytab as oryx/127.27.84.194@KRBTEST.COM (short username oryx)
I20260808 15:49:11.401607 6732 webserver.cc:460] Webserver started at http://127.27.84.194:42325/ using document root <none> and password file <none>
I20260808 15:49:11.402415 6732 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:11.402590 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:11.408524 6732 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260808 15:49:11.411650 6758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:11.413339 6732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:11.413542 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "d4b1753305774f8e81ce34ae635a4df3"
format_stamp: "Formatted at 2026-08-08 15:49:07 on dist-test-slave-twwt"
server_key: "b94d89f7324142f67f433dd1aa3850e9"
server_key_iv: "12191aabfb66c407922f36bf9c74daab"
server_key_version: "encryptionkey@0"
I20260808 15:49:11.414649 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:11.431322 6732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:11.436882 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:11.437160 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:11.438220 6732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:11.439821 6732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:11.439921 6732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:11.440011 6732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:11.440080 6732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:11.479043 6732 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:40679
I20260808 15:49:11.479168 6932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:40679 every 8 connection(s)
I20260808 15:49:11.480780 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:11.490481 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 6732
I20260808 15:49:11.490832 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 5747
Aug 08 15:49:11 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204151, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.252@KRBTEST.COM
Aug 08 15:49:11 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204151, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
Aug 08 15:49:11 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204151, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
Aug 08 15:49:11 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.195@KRBTEST.COM
I20260808 15:49:11.514060 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.484150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:44751 (local address 127.27.84.253:41089)
0808 15:49:11.485958 (+ 1808us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:11.485964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:11.486331 (+ 367us) server_negotiation.cc:407] Connection header received
0808 15:49:11.489033 (+ 2702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:11.489039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:11.489125 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:11.489222 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:11.491868 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.493041 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.498177 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.498966 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.499123 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.504945 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:11.504954 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:11.504961 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:11.505009 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:11.509910 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.510779 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.510784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.510787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.510938 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.511420 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.511426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.511429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.511807 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:11.511965 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:11.513400 (+ 1435us) server_negotiation.cc:299] Negotiation successful
0808 15:49:11.513615 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":1556,"threads_started":1}
I20260808 15:49:11.514477 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.487234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36847 (local address 127.27.84.252:35957)
0808 15:49:11.488112 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:11.488119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:11.488142 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:49:11.489152 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:11.489159 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:11.489273 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:11.489395 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:11.491873 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.493117 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.497221 (+ 4104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.498017 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.498150 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.504315 (+ 6165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:11.504323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:11.504328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:11.504381 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:11.509115 (+ 4734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.510320 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.510327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.510331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.510494 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.511089 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.511095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.511098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.511509 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:11.511667 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:11.513836 (+ 2169us) server_negotiation.cc:299] Negotiation successful
0808 15:49:11.514079 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":611,"threads_started":1}
I20260808 15:49:11.515224 6948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.487082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.27.84.194:36847)
0808 15:49:11.487837 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:11.487863 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:11.488727 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:11.489695 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:11.489708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:11.491335 (+ 1627us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:11.491667 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:11.491675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.493321 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.493327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:11.496990 (+ 3663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:11.497006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.498484 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.498492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:11.498975 (+ 483us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.499947 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:11.499984 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:11.504049 (+ 4065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:11.509327 (+ 5278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:11.509340 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:11.509365 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:11.510102 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:11.510661 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:11.510668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:11.510675 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:11.510950 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:11.512086 (+ 1136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:11.512090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:11.512666 (+ 576us) client_negotiation.cc:769] Sending connection context
0808 15:49:11.513022 (+ 356us) client_negotiation.cc:240] Negotiation successful
0808 15:49:11.513607 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":67,"spinlock_wait_cycles":31872,"thread_start_us":419,"threads_started":1}
I20260808 15:49:11.515638 6944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.484259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:33491 (local address 127.27.84.254:39485)
0808 15:49:11.485561 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:11.485571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:11.486642 (+ 1071us) server_negotiation.cc:407] Connection header received
0808 15:49:11.489033 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:11.489040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:11.489125 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:11.489210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:11.496135 (+ 6925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.497428 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.499842 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.500395 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.500561 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.505511 (+ 4950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:11.505517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:11.505522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:11.505567 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:11.510201 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.511752 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.511758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.511762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.511941 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.513094 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.513101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.513106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.513494 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:11.513647 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:11.514864 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0808 15:49:11.515112 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":1034,"threads_started":1}
I20260808 15:49:11.515692 6947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.484966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:33491)
0808 15:49:11.486491 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:11.486512 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:11.488692 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:11.489706 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:11.489710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:11.495615 (+ 5905us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:11.495944 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:11.495953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.497615 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.497625 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:11.499681 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:11.499699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.500616 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.500623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:11.500854 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.501688 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:11.501711 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:11.505280 (+ 3569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:11.510684 (+ 5404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:11.510691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:11.510695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:11.511362 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:11.512144 (+ 782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:11.512149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:11.512152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:11.512451 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:11.513781 (+ 1330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:11.513785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:11.513934 (+ 149us) client_negotiation.cc:769] Sending connection context
0808 15:49:11.514094 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 15:49:11.515306 (+ 1212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":1141,"thread_start_us":1099,"threads_started":1}
I20260808 15:49:11.516029 6946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.484966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.27.84.194:44751)
0808 15:49:11.486107 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:11.486179 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:11.488692 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:11.489695 (+ 1003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:11.489708 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:11.490686 (+ 978us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:11.491640 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:11.491672 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.496211 (+ 4539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.496232 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:11.497940 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:11.497957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.499448 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.499457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:11.499739 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.501436 (+ 1697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:11.501470 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:11.504668 (+ 3198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:11.510090 (+ 5422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:11.510097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:11.510102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:11.510610 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:11.511083 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:11.511087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:11.511089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:11.511280 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:11.511987 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:11.511999 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:11.512666 (+ 667us) client_negotiation.cc:769] Sending connection context
0808 15:49:11.513022 (+ 356us) client_negotiation.cc:240] Negotiation successful
0808 15:49:11.514650 (+ 1628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":1248,"spinlock_wait_cycles":14720,"thread_start_us":602,"threads_started":1}
I20260808 15:49:11.517199 6937 heartbeater.cc:347] Connected to a master server at 127.27.84.253:41089
I20260808 15:49:11.517346 6935 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39485
I20260808 15:49:11.517251 6934 heartbeater.cc:347] Connected to a master server at 127.27.84.252:35957
I20260808 15:49:11.517606 6937 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:11.517705 6934 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:11.517602 6935 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:11.518846 6934 heartbeater.cc:511] Master 127.27.84.252:35957 requested a full tablet report, sending...
I20260808 15:49:11.518955 6935 heartbeater.cc:511] Master 127.27.84.254:39485 requested a full tablet report, sending...
I20260808 15:49:11.519942 6937 heartbeater.cc:511] Master 127.27.84.253:41089 requested a full tablet report, sending...
I20260808 15:49:11.521018 6156 ts_manager.cc:194] Registered new tserver with Master: d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679)
I20260808 15:49:11.521154 6449 ts_manager.cc:194] Registered new tserver with Master: d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679)
I20260808 15:49:11.521656 5233 ts_manager.cc:194] Re-registered known tserver with Master: d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679)
I20260808 15:49:11.522773 5233 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:33491
I20260808 15:49:11.524094 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:11.493538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.27.84.195:45919)
0808 15:49:11.496710 (+ 3172us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:11.496720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:11.496751 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:49:11.496821 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:11.496830 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:11.497231 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:11.497415 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:11.500915 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.504526 (+ 3611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.506813 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:11.507835 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:11.507996 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:11.516318 (+ 8322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:11.516335 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:11.516354 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:11.516402 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:11.520434 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.521333 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.521341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.521344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.521517 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:11.522294 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:11.522299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:11.522303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:11.522792 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:11.523231 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:11.523424 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 15:49:11.523707 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2996,"thread_start_us":2917,"threads_started":1}
I20260808 15:49:11.525431 5938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:11.613642 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 5747
W20260808 15:49:11.639372 5213 connection.cc:535] server connection from 127.27.84.195:39159 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:49:11.640121 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:45919
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:39485
--webserver_port=37709
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:49:11.876823 6717 heartbeater.cc:503] Master 127.27.84.254:39485 was elected leader, sending a full tablet report...
W20260808 15:49:12.009430 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:12.009783 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:12.010056 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:12.019395 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:12.019577 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:12.019797 6955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:12.031946 6955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:45919
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=37709
--tserver_master_addrs=127.27.84.254:39485,127.27.84.253:41089,127.27.84.252:35957
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:12.033491 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:12.035120 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:12.050599 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:12.050711 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:12.050717 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:12.052014 6955 server_base.cc:1034] running on GCE node
I20260808 15:49:12.052665 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:12.053704 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:12.054929 6955 hybrid_clock.cc:648] HybridClock initialized: now 1786204152054859 us; error 72 us; skew 500 ppm
I20260808 15:49:12.055253 6955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:12 dist-test-slave-twwt krb5kdc[5165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204152, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:12.058519 6955 init.cc:377] Logged in from keytab as oryx/127.27.84.195@KRBTEST.COM (short username oryx)
I20260808 15:49:12.060087 6955 webserver.cc:460] Webserver started at http://127.27.84.195:37709/ using document root <none> and password file <none>
I20260808 15:49:12.060797 6955 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:12.060925 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:12.066565 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260808 15:49:12.069600 6983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:12.071209 6955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:12.071375 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "cb92e1212952499aaed5d509a19d8683"
format_stamp: "Formatted at 2026-08-08 15:49:08 on dist-test-slave-twwt"
server_key: "c7a1d110faaf90d92228c75842762e16"
server_key_iv: "554e287c72eef7d94102708f2e54cb92"
server_key_version: "encryptionkey@0"
I20260808 15:49:12.072865 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:12.092145 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:12.096561 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:12.096810 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:12.097949 6955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:12.099472 6955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:12.099565 6955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:12.099650 6955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:12.099709 6955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:12.135149 6955 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:45919
I20260808 15:49:12.135231 7144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:45919 every 8 connection(s)
I20260808 15:49:12.136716 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:12.143594 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 6955
Aug 08 15:49:12 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204152, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.253@KRBTEST.COM
Aug 08 15:49:12 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204152, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.252@KRBTEST.COM
I20260808 15:49:12.205300 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.139898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:56055 (local address 127.27.84.253:41089)
0808 15:49:12.155255 (+ 15357us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.155266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.159273 (+ 4007us) server_negotiation.cc:407] Connection header received
0808 15:49:12.160829 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.160838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.160941 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.161073 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.165333 (+ 4260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.166906 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.169987 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.170923 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.171082 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.185158 (+ 14076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.185169 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.185177 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.185236 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.191198 (+ 5962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.201155 (+ 9957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.201165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.201172 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.201453 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.202246 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.202255 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.202261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.202891 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.203078 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.204615 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.204892 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15131,"thread_start_us":15749,"threads_started":1}
I20260808 15:49:12.206595 7154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.140507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.27.84.195:56055)
0808 15:49:12.159149 (+ 18642us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.159172 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.160297 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.163769 (+ 3472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.163778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.164640 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:12.165040 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.165056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.167159 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.167168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.169701 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.169722 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.171499 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.171509 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.171937 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.174074 (+ 2137us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:12.174118 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:12.183050 (+ 8932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:12.191505 (+ 8455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:12.191521 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:12.191550 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:12.200933 (+ 9383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:12.201632 (+ 699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:12.201641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:12.201647 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:12.202059 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:12.203247 (+ 1188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:12.203256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:12.203918 (+ 662us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.204330 (+ 412us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.204931 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18462,"mutex_wait_us":41,"thread_start_us":15523,"threads_started":1}
I20260808 15:49:12.209201 7149 heartbeater.cc:347] Connected to a master server at 127.27.84.253:41089
I20260808 15:49:12.209676 7149 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:12.210969 7149 heartbeater.cc:511] Master 127.27.84.253:41089 requested a full tablet report, sending...
Aug 08 15:49:12 dist-test-slave-twwt krb5kdc[5165](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204152, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:12.215562 6156 ts_manager.cc:194] Registered new tserver with Master: cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919)
I20260808 15:49:12.223424 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.140507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.27.84.195:53937)
0808 15:49:12.156642 (+ 16135us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.156700 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.160300 (+ 3600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.162048 (+ 1748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.162068 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.163159 (+ 1091us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:12.164334 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.164369 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.166488 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.166497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.173686 (+ 7189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.173708 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.181841 (+ 8133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.181850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.182134 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.192039 (+ 9905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:12.192074 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:12.209948 (+ 17874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:12.219830 (+ 9882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:12.219837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:12.219841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:12.220448 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:12.221014 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:12.221021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:12.221024 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:12.221245 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:12.222107 (+ 862us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:12.222113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:12.222298 (+ 185us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.222444 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.222847 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15817,"mutex_wait_us":4288,"spinlock_wait_cycles":78592,"thread_start_us":18171,"threads_started":1}
I20260808 15:49:12.224514 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.139903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:53937 (local address 127.27.84.252:35957)
0808 15:49:12.155656 (+ 15753us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.155661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.156858 (+ 1197us) server_negotiation.cc:407] Connection header received
0808 15:49:12.161361 (+ 4503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.161371 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.161483 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.161615 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.164647 (+ 3032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.166199 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.179336 (+ 13137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.180364 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.180507 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.212277 (+ 31770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.212288 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.212297 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.212357 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.219616 (+ 7259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.220628 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.220635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.220638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.220833 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.221403 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.221409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.221412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.221864 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.222033 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.223850 (+ 1817us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.224101 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15638,"thread_start_us":15514,"threads_started":1}
I20260808 15:49:12.225579 7146 heartbeater.cc:347] Connected to a master server at 127.27.84.252:35957
I20260808 15:49:12.225783 7146 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:12.226315 7146 heartbeater.cc:511] Master 127.27.84.252:35957 requested a full tablet report, sending...
I20260808 15:49:12.226411 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.163324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.27.84.253:41089)
0808 15:49:12.205576 (+ 42252us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.205582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.205600 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:49:12.205731 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.205736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.205814 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.205927 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.208565 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.209735 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.211891 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.213539 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.213668 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.218408 (+ 4740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.218415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.218421 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.218466 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.222180 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.223168 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.223174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.223178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.223373 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.223908 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.223913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.223916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.224283 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.224627 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.225770 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.226031 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42144,"thread_start_us":49534,"threads_started":1}
I20260808 15:49:12.228220 6449 ts_manager.cc:194] Registered new tserver with Master: cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919)
I20260808 15:49:12.230187 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.159406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:51823)
0808 15:49:12.159991 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.160011 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.168370 (+ 8359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.179088 (+ 10718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.179097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.179819 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:12.180245 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.180260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.198063 (+ 17803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.198069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.200559 (+ 2490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.200588 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.206082 (+ 5494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.206093 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.206392 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.207831 (+ 1439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:12.207858 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:12.215130 (+ 7272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:12.225366 (+ 10236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:12.225373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:12.225376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:12.226045 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:12.227857 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:12.227864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:12.227868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:12.228059 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:12.229282 (+ 1223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:12.229289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:12.229444 (+ 155us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.229539 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.229767 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":288,"threads_started":1}
I20260808 15:49:12.231005 7164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.159526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51823 (local address 127.27.84.254:39485)
0808 15:49:12.161176 (+ 1650us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.161185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.161209 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:49:12.178731 (+ 17522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.178742 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.178884 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.179038 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.182592 (+ 3554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.197357 (+ 14765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.200937 (+ 3580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.203777 (+ 2840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.207682 (+ 3905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.215371 (+ 7689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.215380 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.215387 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.215432 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.225236 (+ 9804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.226205 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.226213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.226218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.226463 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.228263 (+ 1800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.228267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.228270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.229125 (+ 855us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.230134 (+ 1009us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.230285 (+ 151us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.230534 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1319,"threads_started":1}
I20260808 15:49:12.231669 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.159946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.27.84.254:39485)
0808 15:49:12.168264 (+ 8318us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.168275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.181608 (+ 13333us) server_negotiation.cc:407] Connection header received
0808 15:49:12.184975 (+ 3367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.184984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.185098 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.185233 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.196261 (+ 11028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.197797 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.212183 (+ 14386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.212965 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.213112 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.217464 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.217482 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.217488 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.217537 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.221895 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.228627 (+ 6732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.228632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.228636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.228842 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.229301 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.229306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.229311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.229794 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.230091 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.230761 (+ 670us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.231022 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3003,"thread_start_us":2874,"threads_started":1}
I20260808 15:49:12.231909 7147 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39485
I20260808 15:49:12.232065 7147 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:12.232573 7147 heartbeater.cc:511] Master 127.27.84.254:39485 requested a full tablet report, sending...
I20260808 15:49:12.233700 5233 ts_manager.cc:194] Re-registered known tserver with Master: cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919)
I20260808 15:49:12.234915 5233 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:51823
I20260808 15:49:12.237910 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.181800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.27.84.252:35957)
0808 15:49:12.200634 (+ 18834us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.200643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.212201 (+ 11558us) server_negotiation.cc:407] Connection header received
0808 15:49:12.212273 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.212280 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.212364 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.212492 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.220532 (+ 8040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.221941 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.224138 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.224964 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.225079 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.230955 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.230963 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.230970 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.231014 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.234677 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.235548 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.235554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.235557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.235807 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.236285 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.236290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.236293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.236645 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.237014 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.237230 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.237519 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18642,"thread_start_us":18840,"threads_started":1}
I20260808 15:49:12.236510 5233 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:12.245210 5233 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:12.305657 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.283069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.27.84.195:45919)
0808 15:49:12.294024 (+ 10955us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.294039 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.295394 (+ 1355us) server_negotiation.cc:407] Connection header received
0808 15:49:12.295658 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.295667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.295983 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.296177 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:12.297056 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.298834 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.302310 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.304175 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.304350 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.304631 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.305050 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.305236 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10231,"thread_start_us":10406,"threads_started":1}
I20260808 15:49:12.305704 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.282649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45919 (local address 127.0.0.1:39878)
0808 15:49:12.295209 (+ 12560us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.295236 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.295433 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.296201 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.296209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.296245 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:12.296785 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.296802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.299032 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.299039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.302078 (+ 3039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.302095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.304520 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.304528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.304792 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.304845 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.304977 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.305155 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12357,"mutex_wait_us":4,"thread_start_us":12095,"threads_started":1}
I20260808 15:49:12.309087 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.283488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.27.84.193:33515)
0808 15:49:12.294513 (+ 11025us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.294528 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.299954 (+ 5426us) server_negotiation.cc:407] Connection header received
0808 15:49:12.300206 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.300218 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.300529 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.300770 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:12.301915 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.303386 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.307057 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.308007 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.308169 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.308400 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.308619 (+ 219us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.308782 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10866,"thread_start_us":10740,"threads_started":1}
I20260808 15:49:12.309162 7194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.283269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:33515 (local address 127.0.0.1:44614)
0808 15:49:12.299795 (+ 16526us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.299818 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.299987 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.300976 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.300979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.300999 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:12.301557 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.301573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.303676 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.303684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.306886 (+ 3202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.306902 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.308230 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.308236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.308404 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.308444 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.308562 (+ 118us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.308693 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16381,"mutex_wait_us":3,"thread_start_us":17004,"threads_started":1}
I20260808 15:49:12.309638 7053 tablet_service.cc:1467] Processing CreateTablet for tablet d5b85569dc87430a9e1a27c45437837b (DEFAULT_TABLE table=test-table [id=f121f26d9eb44dc4bbb2363a1915d571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:12.311432 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b85569dc87430a9e1a27c45437837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:12.313416 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.285223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:40679 (local address 127.0.0.1:55336)
0808 15:49:12.300557 (+ 15334us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.300584 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.300729 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.301342 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.301349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.301379 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:12.302021 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.302032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.304703 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.304709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.306880 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.306898 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.312529 (+ 5631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.312535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.312738 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.312770 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.312884 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.313023 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15212,"thread_start_us":14236,"threads_started":1}
I20260808 15:49:12.313465 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.285958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.27.84.194:40679)
0808 15:49:12.299275 (+ 13317us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.299285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.300730 (+ 1445us) server_negotiation.cc:407] Connection header received
0808 15:49:12.300887 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.300894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.301123 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.301319 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:12.302186 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.303802 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.310722 (+ 6920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.312267 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.312428 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.312623 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.312945 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.313127 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12948,"thread_start_us":12826,"threads_started":1}
I20260808 15:49:12.313118 6624 tablet_service.cc:1467] Processing CreateTablet for tablet d5b85569dc87430a9e1a27c45437837b (DEFAULT_TABLE table=test-table [id=f121f26d9eb44dc4bbb2363a1915d571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:12.314563 6624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b85569dc87430a9e1a27c45437837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:12.317768 6832 tablet_service.cc:1467] Processing CreateTablet for tablet d5b85569dc87430a9e1a27c45437837b (DEFAULT_TABLE table=test-table [id=f121f26d9eb44dc4bbb2363a1915d571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:12.319335 6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5b85569dc87430a9e1a27c45437837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:12.325637 7197 tablet_bootstrap.cc:492] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Bootstrap starting.
I20260808 15:49:12.330847 7197 tablet_bootstrap.cc:654] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:12.332115 7197 log.cc:824] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:12.334694 7205 tablet_bootstrap.cc:492] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Bootstrap starting.
I20260808 15:49:12.336154 7197 tablet_bootstrap.cc:492] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: No bootstrap required, opened a new log
I20260808 15:49:12.336519 7197 ts_tablet_manager.cc:1397] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260808 15:49:12.337639 7206 tablet_bootstrap.cc:492] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Bootstrap starting.
I20260808 15:49:12.339351 7205 tablet_bootstrap.cc:654] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:12.340932 7205 log.cc:824] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:12.342031 7206 tablet_bootstrap.cc:654] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:12.343626 7206 log.cc:824] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:12.344941 7205 tablet_bootstrap.cc:492] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: No bootstrap required, opened a new log
I20260808 15:49:12.345353 7205 ts_tablet_manager.cc:1397] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260808 15:49:12.346877 7206 tablet_bootstrap.cc:492] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: No bootstrap required, opened a new log
I20260808 15:49:12.347177 7206 ts_tablet_manager.cc:1397] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260808 15:49:12.348579 7197 raft_consensus.cc:348] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.349128 7197 raft_consensus.cc:374] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:12.349269 7197 raft_consensus.cc:729] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb92e1212952499aaed5d509a19d8683, State: Initialized, Role: FOLLOWER
I20260808 15:49:12.350085 7197 consensus_queue.cc:260] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.352221 7197 ts_tablet_manager.cc:1428] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Time spent starting tablet: real 0.016s user 0.000s sys 0.015s
I20260808 15:49:12.352567 7147 heartbeater.cc:503] Master 127.27.84.254:39485 was elected leader, sending a full tablet report...
I20260808 15:49:12.357159 7205 raft_consensus.cc:348] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.357692 7205 raft_consensus.cc:374] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:12.357870 7205 raft_consensus.cc:729] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ec6fb7ca846df819e9a930eff7e5f, State: Initialized, Role: FOLLOWER
I20260808 15:49:12.358567 7205 consensus_queue.cc:260] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.360236 7206 raft_consensus.cc:348] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.360752 7206 raft_consensus.cc:374] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:12.360945 7206 raft_consensus.cc:729] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b1753305774f8e81ce34ae635a4df3, State: Initialized, Role: FOLLOWER
I20260808 15:49:12.361737 7206 consensus_queue.cc:260] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.362861 7205 ts_tablet_manager.cc:1428] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
I20260808 15:49:12.365715 7206 ts_tablet_manager.cc:1428] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260808 15:49:12.366088 6935 heartbeater.cc:503] Master 127.27.84.254:39485 was elected leader, sending a full tablet report...
W20260808 15:49:12.389791 7150 tablet.cc:2369] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:49:12.484567 6938 tablet.cc:2369] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:12.522078 7210 raft_consensus.cc:482] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:12.522462 7210 raft_consensus.cc:504] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.524576 7210 leader_election.cc:288] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515), d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679)
I20260808 15:49:12.533382 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.525312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:56109 (local address 127.27.84.194:40679)
0808 15:49:12.525529 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.525533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.525549 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:49:12.525611 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.525614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.525671 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.525780 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:12.526262 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.527744 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.530854 (+ 3110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.532225 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.532425 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.532492 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.532967 (+ 475us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.533105 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:49:12.533378 7193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.525253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:54961 (local address 127.27.84.193:33515)
0808 15:49:12.525556 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.525560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.525577 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:49:12.525671 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.525677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.525817 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.525926 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:12.526440 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.527781 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.530886 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.532222 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.532410 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.532489 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.532967 (+ 478us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.533105 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260808 15:49:12.533466 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.524869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:33515 (local address 127.27.84.195:54961)
0808 15:49:12.525096 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.525119 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.525282 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.526032 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.526037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.526058 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:12.526265 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.526274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.527921 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.527925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.530666 (+ 2741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.530689 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.532409 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.532415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.532813 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.532841 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.532919 (+ 78us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.533142 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":278}
I20260808 15:49:12.533481 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.524983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:40679 (local address 127.27.84.195:56109)
0808 15:49:12.525206 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:12.525227 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:12.525342 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:12.525804 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:12.525809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:12.525842 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:12.526070 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.526082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.527918 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.527924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.530687 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:12.530697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.532409 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.532415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:12.532738 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.532797 (+ 59us) client_negotiation.cc:769] Sending connection context
0808 15:49:12.532913 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:49:12.533136 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":291}
I20260808 15:49:12.534747 6648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b85569dc87430a9e1a27c45437837b" candidate_uuid: "cb92e1212952499aaed5d509a19d8683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" is_pre_election: true
I20260808 15:49:12.534932 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b85569dc87430a9e1a27c45437837b" candidate_uuid: "cb92e1212952499aaed5d509a19d8683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1753305774f8e81ce34ae635a4df3" is_pre_election: true
I20260808 15:49:12.535323 6648 raft_consensus.cc:2454] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb92e1212952499aaed5d509a19d8683 in term 0.
I20260808 15:49:12.535449 6868 raft_consensus.cc:2454] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb92e1212952499aaed5d509a19d8683 in term 0.
I20260808 15:49:12.536209 6987 leader_election.cc:302] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cb92e1212952499aaed5d509a19d8683, d4b1753305774f8e81ce34ae635a4df3, ee5ec6fb7ca846df819e9a930eff7e5f; no voters:
I20260808 15:49:12.536630 7210 raft_consensus.cc:2781] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:12.536799 7210 raft_consensus.cc:482] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:12.536991 7210 raft_consensus.cc:3037] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:12.539589 7210 raft_consensus.cc:504] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.540302 7210 leader_election.cc:288] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [CANDIDATE]: Term 1 election: Requested vote from peers ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515), d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679)
I20260808 15:49:12.540779 6648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b85569dc87430a9e1a27c45437837b" candidate_uuid: "cb92e1212952499aaed5d509a19d8683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f"
I20260808 15:49:12.540819 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5b85569dc87430a9e1a27c45437837b" candidate_uuid: "cb92e1212952499aaed5d509a19d8683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1753305774f8e81ce34ae635a4df3"
I20260808 15:49:12.541036 6868 raft_consensus.cc:3037] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:12.541097 6648 raft_consensus.cc:3037] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:12.544626 6868 raft_consensus.cc:2454] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb92e1212952499aaed5d509a19d8683 in term 1.
I20260808 15:49:12.544626 6648 raft_consensus.cc:2454] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb92e1212952499aaed5d509a19d8683 in term 1.
I20260808 15:49:12.545099 6987 leader_election.cc:302] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: cb92e1212952499aaed5d509a19d8683, d4b1753305774f8e81ce34ae635a4df3, ee5ec6fb7ca846df819e9a930eff7e5f; no voters:
I20260808 15:49:12.545564 7210 raft_consensus.cc:2781] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:12.545998 7210 raft_consensus.cc:686] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 1 LEADER]: Becoming Leader. State: Replica: cb92e1212952499aaed5d509a19d8683, State: Running, Role: LEADER
I20260808 15:49:12.546595 7210 consensus_queue.cc:237] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } }
I20260808 15:49:12.552379 5234 catalog_manager.cc:5495] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb92e1212952499aaed5d509a19d8683 (127.27.84.195). New cstate: current_term: 1 leader_uuid: "cb92e1212952499aaed5d509a19d8683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } health_report { overall_health: UNKNOWN } } }
W20260808 15:49:12.567888 6729 tablet.cc:2369] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:12.601095 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.595187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.27.84.195:45919)
0808 15:49:12.595390 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.595394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.595542 (+ 148us) server_negotiation.cc:407] Connection header received
0808 15:49:12.595780 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.595785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.595900 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.596050 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:12.596403 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.597131 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.598749 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.599177 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.599280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.599835 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:12.600421 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:12.600566 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.600653 (+ 87us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.600737 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:49:12.610868 6648 raft_consensus.cc:1261] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Refusing update from remote peer cb92e1212952499aaed5d509a19d8683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:12.610865 6868 raft_consensus.cc:1261] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Refusing update from remote peer cb92e1212952499aaed5d509a19d8683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:12.612037 7213 consensus_queue.cc:1035] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:12.612609 7210 consensus_queue.cc:1035] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:12.646420 5234 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:12.646800 5234 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:12.654073 5234 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b173194b048f444294d077c949db5fc6: Sending DeleteTablet for 3 replicas of tablet d5b85569dc87430a9e1a27c45437837b
I20260808 15:49:12.655336 7053 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b85569dc87430a9e1a27c45437837b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:12 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:39878
I20260808 15:49:12.655725 6624 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b85569dc87430a9e1a27c45437837b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:12 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:44614
I20260808 15:49:12.655937 6832 tablet_service.cc:1514] Processing DeleteTablet for tablet d5b85569dc87430a9e1a27c45437837b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:12 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:55336
I20260808 15:49:12.667445 7234 tablet_replica.cc:330] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: stopping tablet replica
I20260808 15:49:12.669281 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 5188
I20260808 15:49:12.670517 5281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:12.670567 7234 raft_consensus.cc:2229] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:12.671375 7234 raft_consensus.cc:2258] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:12.678913 7234 ts_tablet_manager.cc:1905] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:12.682214 7233 tablet_replica.cc:330] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: stopping tablet replica
I20260808 15:49:12.683071 7235 tablet_replica.cc:330] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: stopping tablet replica
I20260808 15:49:12.683634 7235 raft_consensus.cc:2229] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:12.684341 7235 raft_consensus.cc:2258] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:12.684320 7233 raft_consensus.cc:2229] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:12.686903 7235 ts_tablet_manager.cc:1905] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:12.694643 7233 raft_consensus.cc:2258] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:12.694875 7235 ts_tablet_manager.cc:1918] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:12.695056 7235 log.cc:1196] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/d5b85569dc87430a9e1a27c45437837b
I20260808 15:49:12.696017 7235 ts_tablet_manager.cc:1939] T d5b85569dc87430a9e1a27c45437837b P d4b1753305774f8e81ce34ae635a4df3: Deleting consensus metadata
I20260808 15:49:12.697544 7233 ts_tablet_manager.cc:1905] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:12.702008 7234 ts_tablet_manager.cc:1918] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:12.704761 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.659038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.27.84.253:41089)
0808 15:49:12.659278 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.659286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.670882 (+ 11596us) server_negotiation.cc:407] Connection header received
0808 15:49:12.670986 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.670995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.671112 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.671233 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.674384 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.676533 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.683361 (+ 6828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.684088 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.684228 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.694405 (+ 10177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.694415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.694422 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.694481 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.698479 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.699382 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.699387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.699391 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.699550 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.702310 (+ 2760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.702316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.702320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.702857 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.703154 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.703916 (+ 762us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.704202 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:49:12.708230 7233 ts_tablet_manager.cc:1918] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:12.708429 7233 log.cc:1196] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/d5b85569dc87430a9e1a27c45437837b
I20260808 15:49:12.708591 7234 log.cc:1196] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/d5b85569dc87430a9e1a27c45437837b
I20260808 15:49:12.709030 7233 ts_tablet_manager.cc:1939] T d5b85569dc87430a9e1a27c45437837b P cb92e1212952499aaed5d509a19d8683: Deleting consensus metadata
I20260808 15:49:12.711097 7234 ts_tablet_manager.cc:1939] T d5b85569dc87430a9e1a27c45437837b P ee5ec6fb7ca846df819e9a930eff7e5f: Deleting consensus metadata
I20260808 15:49:12.724815 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.668433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.27.84.252:35957)
0808 15:49:12.668632 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.668638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.705668 (+ 37030us) server_negotiation.cc:407] Connection header received
0808 15:49:12.706000 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.706010 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.706150 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.706259 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.708603 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.709913 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.712109 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.712945 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.713072 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.717091 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.717099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.717105 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.717156 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.721554 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.722415 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.722422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.722426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.722599 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.723076 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.723082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.723088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.723504 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.723795 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.724152 (+ 357us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.724420 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:49:12.831028 5211 catalog_manager.cc:4841] TS d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679): tablet d5b85569dc87430a9e1a27c45437837b (table test-table [id=f121f26d9eb44dc4bbb2363a1915d571]) successfully deleted
I20260808 15:49:12.831118 5214 catalog_manager.cc:4841] TS cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919): tablet d5b85569dc87430a9e1a27c45437837b (table test-table [id=f121f26d9eb44dc4bbb2363a1915d571]) successfully deleted
I20260808 15:49:12.831061 5213 catalog_manager.cc:4841] TS ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515): tablet d5b85569dc87430a9e1a27c45437837b (table test-table [id=f121f26d9eb44dc4bbb2363a1915d571]) successfully deleted
I20260808 15:49:12.832194 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 5188
I20260808 15:49:12.895778 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.875211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.27.84.253:41089)
0808 15:49:12.875500 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.875510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.876554 (+ 1044us) server_negotiation.cc:407] Connection header received
0808 15:49:12.877598 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.877603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.877675 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.877794 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.880413 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.881587 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.883297 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.884046 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.884158 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.888464 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.888472 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.888478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.888531 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.892429 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.893540 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.893546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.893550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.893726 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.894142 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.894149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.894155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.894543 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.894870 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.895225 (+ 355us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.895454 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:49:12.896385 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:12.876529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.27.84.252:35957)
0808 15:49:12.876726 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:12.876732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:12.877204 (+ 472us) server_negotiation.cc:407] Connection header received
0808 15:49:12.877557 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:12.877564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:12.877694 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:12.877807 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:12.881794 (+ 3987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.883104 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.884802 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:12.885452 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:12.885608 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:12.889201 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:12.889210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:12.889216 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:12.889264 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:12.892899 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.893786 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.893792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.893796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.893948 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:12.894361 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:12.894366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:12.894370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:12.894777 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:12.895019 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:12.895769 (+ 750us) server_negotiation.cc:299] Negotiation successful
0808 15:49:12.896010 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:49:13.634348 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:13.633431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:55581)
0808 15:49:13.633959 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:13.634111 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":287,"threads_started":1}
W20260808 15:49:13.634821 7147 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:39485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:13.638144 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:13.637461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:50235)
0808 15:49:13.637870 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:13.637988 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":196,"threads_started":1}
W20260808 15:49:13.638590 6717 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:39485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:13.640437 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:13.639609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:39233)
0808 15:49:13.640141 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:13.640291 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":277,"threads_started":1}
W20260808 15:49:13.640853 6935 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:39485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:14.154515 7254 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b173194b048f444294d077c949db5fc6)
I20260808 15:49:14.154549 7255 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b173194b048f444294d077c949db5fc6)
I20260808 15:49:14.154790 7254 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } }
I20260808 15:49:14.154910 7255 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } }
I20260808 15:49:14.155720 7254 leader_election.cc:288] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b173194b048f444294d077c949db5fc6 (127.27.84.254:39485), fa99b51165794de98294ac7c51b86512 (127.27.84.253:41089)
I20260808 15:49:14.157022 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.155915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57926)
0808 15:49:14.156487 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.156693 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"threads_started":1}
I20260808 15:49:14.157110 7255 leader_election.cc:288] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b173194b048f444294d077c949db5fc6 (127.27.84.254:39485), f83ced808e574044a6d122cfe7d700ab (127.27.84.252:35957)
I20260808 15:49:14.158131 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.156674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57938)
0808 15:49:14.157622 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.157834 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":661,"threads_started":1}
W20260808 15:49:14.159246 6423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
W20260808 15:49:14.160074 6134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:14.161545 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.160871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57946)
0808 15:49:14.161100 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.161291 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 15:49:14.161844 6423 leader_election.cc:334] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b173194b048f444294d077c949db5fc6 (127.27.84.254:39485): Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:14.163755 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.163118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57948)
0808 15:49:14.163331 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.163520 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 15:49:14.164041 6134 leader_election.cc:334] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b173194b048f444294d077c949db5fc6 (127.27.84.254:39485): Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:14.165182 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.155944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41089 (local address 127.0.0.1:58170)
0808 15:49:14.157386 (+ 1442us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.157406 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.157618 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.158334 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.158339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.158369 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.158654 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.158670 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.160182 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.160187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.162349 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.162369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.164357 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.164365 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.164580 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.164608 (+ 28us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.164721 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.164877 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"thread_start_us":274,"threads_started":1}
I20260808 15:49:14.165273 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.156174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.27.84.253:41089)
0808 15:49:14.156852 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.156859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.157499 (+ 640us) server_negotiation.cc:407] Connection header received
0808 15:49:14.157806 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.157829 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.157939 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.158016 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.158902 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.159985 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.162568 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.164117 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.164285 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.164523 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.164766 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.164914 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":536,"thread_start_us":337,"threads_started":1}
I20260808 15:49:14.166000 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f83ced808e574044a6d122cfe7d700ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa99b51165794de98294ac7c51b86512" is_pre_election: true
I20260808 15:49:14.166270 6179 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f83ced808e574044a6d122cfe7d700ab in term 1.
I20260808 15:49:14.166850 6424 leader_election.cc:302] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f83ced808e574044a6d122cfe7d700ab, fa99b51165794de98294ac7c51b86512; no voters: b173194b048f444294d077c949db5fc6
I20260808 15:49:14.167235 7254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260808 15:49:14.167384 7254 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader b173194b048f444294d077c949db5fc6)
I20260808 15:49:14.167488 7254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:49:14.167968 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.157857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:35957 (local address 127.0.0.1:43312)
0808 15:49:14.158432 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.158452 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.158566 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.161536 (+ 2970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.161544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.161583 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.162054 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.162070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.163814 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.163819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.165666 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.165684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.167271 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.167278 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.167516 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.167573 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.167674 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.167780 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":1856,"threads_started":1}
I20260808 15:49:14.168437 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.158378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.27.84.252:35957)
0808 15:49:14.161095 (+ 2717us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.161104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.161127 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:49:14.161196 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.161208 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.161325 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.162115 (+ 790us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.162392 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.163659 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.165838 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.167051 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.167229 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.167363 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.167719 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.167897 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2149,"thread_start_us":2177,"threads_started":1}
I20260808 15:49:14.169122 6462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa99b51165794de98294ac7c51b86512" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f83ced808e574044a6d122cfe7d700ab" is_pre_election: true
I20260808 15:49:14.170498 7254 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } }
I20260808 15:49:14.171036 6462 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa99b51165794de98294ac7c51b86512 in current term 2: Already voted for candidate f83ced808e574044a6d122cfe7d700ab in this term.
I20260808 15:49:14.171849 7254 leader_election.cc:288] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 2 election: Requested vote from peers b173194b048f444294d077c949db5fc6 (127.27.84.254:39485), fa99b51165794de98294ac7c51b86512 (127.27.84.253:41089)
I20260808 15:49:14.171974 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.171501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57962)
0808 15:49:14.171665 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.171796 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 15:49:14.172238 6134 leader_election.cc:302] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa99b51165794de98294ac7c51b86512; no voters: b173194b048f444294d077c949db5fc6, f83ced808e574044a6d122cfe7d700ab
I20260808 15:49:14.172665 7255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:49:14.172572 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f83ced808e574044a6d122cfe7d700ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fa99b51165794de98294ac7c51b86512"
I20260808 15:49:14.173290 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.172799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57970)
0808 15:49:14.173033 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.173141 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260808 15:49:14.173475 6423 leader_election.cc:334] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b173194b048f444294d077c949db5fc6 (127.27.84.254:39485): Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
I20260808 15:49:14.175278 7255 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:49:14.177716 6179 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f83ced808e574044a6d122cfe7d700ab in term 2.
I20260808 15:49:14.178579 6424 leader_election.cc:302] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f83ced808e574044a6d122cfe7d700ab, fa99b51165794de98294ac7c51b86512; no voters: b173194b048f444294d077c949db5fc6
I20260808 15:49:14.179131 7254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:49:14.179668 7254 raft_consensus.cc:686] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [term 2 LEADER]: Becoming Leader. State: Replica: f83ced808e574044a6d122cfe7d700ab, State: Running, Role: LEADER
I20260808 15:49:14.180085 7254 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } }
I20260808 15:49:14.181751 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83ced808e574044a6d122cfe7d700ab. Latest consensus state: current_term: 2 leader_uuid: "f83ced808e574044a6d122cfe7d700ab" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:14.182080 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:14.183012 7268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:14.186957 7268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:14.188046 7268 catalog_manager.cc:1259] Loaded cluster ID: cf9b92ee36964699ba0615c5f9a7497b
I20260808 15:49:14.188138 7268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:14.190104 7268 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:14.191026 7268 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Loaded TSK: 0
I20260808 15:49:14.191962 7268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:49:14.245645 6449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:14.253293 6449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:14.258332 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.257831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:57992)
0808 15:49:14.258041 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.258138 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:49:14.258293 6179 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [term 2 FOLLOWER]: Refusing update from remote peer f83ced808e574044a6d122cfe7d700ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260808 15:49:14.259279 7254 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260808 15:49:14.259418 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.259020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:58002)
0808 15:49:14.259200 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.259289 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 15:49:14.259717 6423 consensus_peers.cc:487] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab -> Peer b173194b048f444294d077c949db5fc6 (127.27.84.254:39485): Couldn't send request to peer b173194b048f444294d077c949db5fc6. Status: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260808 15:49:14.262559 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83ced808e574044a6d122cfe7d700ab. Latest consensus state: current_term: 2 leader_uuid: "f83ced808e574044a6d122cfe7d700ab" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:14.263001 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:49:14.264344 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f83ced808e574044a6d122cfe7d700ab" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:14.264829 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:14.265383 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f83ced808e574044a6d122cfe7d700ab" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "b173194b048f444294d077c949db5fc6" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39485 } } peers { permanent_uuid: "fa99b51165794de98294ac7c51b86512" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41089 } attrs { promote: false } } peers { permanent_uuid: "f83ced808e574044a6d122cfe7d700ab" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 35957 } attrs { promote: false } } }
I20260808 15:49:14.265691 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa99b51165794de98294ac7c51b86512 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:49:14.293715 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.274883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.27.84.195:45919)
0808 15:49:14.287267 (+ 12384us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.287274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.287297 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:49:14.287378 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.287385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.287474 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.287804 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.288399 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.289290 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.291268 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.292493 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.292631 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.292766 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.293230 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.293375 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12217,"thread_start_us":12677,"threads_started":1}
I20260808 15:49:14.293771 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.274336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45919 (local address 127.0.0.1:33628)
0808 15:49:14.274934 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.274951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.275065 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.287823 (+ 12758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.287828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.287854 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.288227 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.288237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.289436 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.289443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.291058 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.291074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.292905 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.292909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.293047 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.293085 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.293196 (+ 111us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.293334 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":2,"thread_start_us":669,"threads_started":1}
I20260808 15:49:14.294754 7053 tablet_service.cc:1467] Processing CreateTablet for tablet 47af19bdfc61495db72b9a7bc5357346 (DEFAULT_TABLE table=test-table [id=679d654faeea4b6ba2aa0d73c87cb520]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:14.295316 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47af19bdfc61495db72b9a7bc5357346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:14.295789 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.273960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:40679 (local address 127.0.0.1:43602)
0808 15:49:14.274104 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.274120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.274227 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.288351 (+ 14124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.288360 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.288397 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.288794 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.288810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.290883 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.290890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.293082 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.293098 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.294875 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.294880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.295134 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.295160 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.295279 (+ 119us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.295436 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":89}
I20260808 15:49:14.296108 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.274415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43602 (local address 127.27.84.194:40679)
0808 15:49:14.287881 (+ 13466us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.287890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.287917 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:49:14.288011 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.288019 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.288123 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.288989 (+ 866us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.289210 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.290697 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.293354 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.294459 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.294640 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.294748 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.295313 (+ 565us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.295587 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13305,"thread_start_us":13115,"threads_started":1}
I20260808 15:49:14.297422 6832 tablet_service.cc:1467] Processing CreateTablet for tablet 47af19bdfc61495db72b9a7bc5357346 (DEFAULT_TABLE table=test-table [id=679d654faeea4b6ba2aa0d73c87cb520]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:14.297940 6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47af19bdfc61495db72b9a7bc5357346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:14.299449 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.273426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:33515 (local address 127.0.0.1:57716)
0808 15:49:14.273739 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.273769 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.273882 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.291384 (+ 17502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.291389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.291417 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.291630 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.291640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.294872 (+ 3232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.294878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.296719 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.296735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.298662 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.298670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.298876 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.298910 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.298986 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.299132 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":2}
I20260808 15:49:14.300454 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.273575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57716 (local address 127.27.84.193:33515)
0808 15:49:14.291011 (+ 17436us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.291019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.291041 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:49:14.291117 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.291124 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.291221 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.291340 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.291882 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.293186 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.297230 (+ 4044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.298446 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.299802 (+ 1356us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.299916 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.299971 (+ 55us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.300122 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17305,"thread_start_us":17223,"threads_started":1}
I20260808 15:49:14.301297 6624 tablet_service.cc:1467] Processing CreateTablet for tablet 47af19bdfc61495db72b9a7bc5357346 (DEFAULT_TABLE table=test-table [id=679d654faeea4b6ba2aa0d73c87cb520]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:14.301793 6624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47af19bdfc61495db72b9a7bc5357346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:14.302402 7282 tablet_bootstrap.cc:492] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Bootstrap starting.
I20260808 15:49:14.304260 7283 tablet_bootstrap.cc:492] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Bootstrap starting.
I20260808 15:49:14.304411 7282 tablet_bootstrap.cc:654] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:14.306695 7283 tablet_bootstrap.cc:654] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:14.309145 7289 tablet_bootstrap.cc:492] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Bootstrap starting.
I20260808 15:49:14.309478 7283 tablet_bootstrap.cc:492] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: No bootstrap required, opened a new log
I20260808 15:49:14.309676 7283 ts_tablet_manager.cc:1397] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260808 15:49:14.310774 7289 tablet_bootstrap.cc:654] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:14.310797 7283 raft_consensus.cc:348] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.311087 7283 raft_consensus.cc:374] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:14.311187 7283 raft_consensus.cc:729] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b1753305774f8e81ce34ae635a4df3, State: Initialized, Role: FOLLOWER
I20260808 15:49:14.311452 7283 consensus_queue.cc:260] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.312531 7282 tablet_bootstrap.cc:492] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: No bootstrap required, opened a new log
I20260808 15:49:14.312800 7289 tablet_bootstrap.cc:492] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: No bootstrap required, opened a new log
I20260808 15:49:14.312804 7282 ts_tablet_manager.cc:1397] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260808 15:49:14.312978 7289 ts_tablet_manager.cc:1397] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260808 15:49:14.313680 7289 raft_consensus.cc:348] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.313969 7289 raft_consensus.cc:374] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:14.314070 7289 raft_consensus.cc:729] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee5ec6fb7ca846df819e9a930eff7e5f, State: Initialized, Role: FOLLOWER
I20260808 15:49:14.313915 7282 raft_consensus.cc:348] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.314211 7282 raft_consensus.cc:374] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:14.314325 7282 raft_consensus.cc:729] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb92e1212952499aaed5d509a19d8683, State: Initialized, Role: FOLLOWER
I20260808 15:49:14.314320 7289 consensus_queue.cc:260] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.314738 7282 consensus_queue.cc:260] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.318713 7283 ts_tablet_manager.cc:1428] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20260808 15:49:14.318878 7282 ts_tablet_manager.cc:1428] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260808 15:49:14.323132 7289 ts_tablet_manager.cc:1428] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260808 15:49:14.323659 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.321289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:44125)
0808 15:49:14.322463 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.322660 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1046,"thread_start_us":898,"threads_started":1}
I20260808 15:49:14.325536 7146 heartbeater.cc:503] Master 127.27.84.252:35957 was elected leader, sending a full tablet report...
I20260808 15:49:14.329492 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.325901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:57413)
0808 15:49:14.329118 (+ 3217us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.329271 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2784}
I20260808 15:49:14.331518 6934 heartbeater.cc:503] Master 127.27.84.252:35957 was elected leader, sending a full tablet report...
I20260808 15:49:14.331766 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.319917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:40907)
0808 15:49:14.331184 (+ 11267us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.331403 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11132,"thread_start_us":10965,"threads_started":1}
I20260808 15:49:14.332263 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.322897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:58351)
0808 15:49:14.331747 (+ 8850us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.331982 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8627,"thread_start_us":8473,"threads_started":1}
I20260808 15:49:14.333247 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.332824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:59147)
0808 15:49:14.333021 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.333109 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 15:49:14.333592 6715 heartbeater.cc:503] Master 127.27.84.252:35957 was elected leader, sending a full tablet report...
I20260808 15:49:14.333752 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.333117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:60747)
0808 15:49:14.333514 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.333606 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267}
I20260808 15:49:14.345827 7295 raft_consensus.cc:482] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:14.346045 7295 raft_consensus.cc:504] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.347854 7295 leader_election.cc:288] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679), cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919)
I20260808 15:49:14.356668 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.348339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:52349 (local address 127.27.84.195:45919)
0808 15:49:14.348536 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.348544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.348986 (+ 442us) server_negotiation.cc:407] Connection header received
0808 15:49:14.349127 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.349132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.349220 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.349307 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.350025 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.351198 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.354489 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.355594 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.355727 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.355787 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.356249 (+ 462us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.356413 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260808 15:49:14.356861 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.348057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:40679 (local address 127.27.84.193:49037)
0808 15:49:14.348616 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.348644 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.348826 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.349265 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.349268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.349319 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.349816 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.349837 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.351367 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.351373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.354241 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.354260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.356039 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.356044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.356233 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.356269 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.356382 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.356522 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":211,"spinlock_wait_cycles":5760}
I20260808 15:49:14.356946 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.348192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45919 (local address 127.27.84.193:52349)
0808 15:49:14.348868 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.348888 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:14.349004 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:14.349329 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:14.349333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:14.349358 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:14.349863 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.349874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.351367 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.351373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.354241 (+ 2868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:14.354260 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.355802 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.355810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:14.356036 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.356076 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:49:14.356177 (+ 101us) client_negotiation.cc:240] Negotiation successful
0808 15:49:14.356300 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":260,"spinlock_wait_cycles":768,"thread_start_us":357,"threads_started":1}
I20260808 15:49:14.357182 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.348184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:49037 (local address 127.27.84.194:40679)
0808 15:49:14.348437 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.348445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.348759 (+ 314us) server_negotiation.cc:407] Connection header received
0808 15:49:14.349016 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.349023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.349130 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.349233 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:14.350016 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.351187 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.354756 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.355801 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.355979 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.356056 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.356439 (+ 383us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.356585 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:49:14.357822 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47af19bdfc61495db72b9a7bc5357346" candidate_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1753305774f8e81ce34ae635a4df3" is_pre_election: true
I20260808 15:49:14.358081 6868 raft_consensus.cc:2454] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5ec6fb7ca846df819e9a930eff7e5f in term 0.
I20260808 15:49:14.358003 7081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47af19bdfc61495db72b9a7bc5357346" candidate_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb92e1212952499aaed5d509a19d8683" is_pre_election: true
I20260808 15:49:14.358505 7081 raft_consensus.cc:2454] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee5ec6fb7ca846df819e9a930eff7e5f in term 0.
I20260808 15:49:14.358691 6560 leader_election.cc:302] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b1753305774f8e81ce34ae635a4df3, ee5ec6fb7ca846df819e9a930eff7e5f; no voters:
I20260808 15:49:14.359179 7295 raft_consensus.cc:2781] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:14.359349 7295 raft_consensus.cc:482] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:14.359463 7295 raft_consensus.cc:3037] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:14.361681 7295 raft_consensus.cc:504] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.362380 7295 leader_election.cc:288] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [CANDIDATE]: Term 1 election: Requested vote from peers d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679), cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919)
I20260808 15:49:14.362953 6868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47af19bdfc61495db72b9a7bc5357346" candidate_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4b1753305774f8e81ce34ae635a4df3"
I20260808 15:49:14.363004 7081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47af19bdfc61495db72b9a7bc5357346" candidate_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb92e1212952499aaed5d509a19d8683"
I20260808 15:49:14.363209 7081 raft_consensus.cc:3037] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:14.363209 6868 raft_consensus.cc:3037] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:14.366384 6868 raft_consensus.cc:2454] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5ec6fb7ca846df819e9a930eff7e5f in term 1.
I20260808 15:49:14.366384 7081 raft_consensus.cc:2454] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee5ec6fb7ca846df819e9a930eff7e5f in term 1.
I20260808 15:49:14.366865 6560 leader_election.cc:302] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4b1753305774f8e81ce34ae635a4df3, ee5ec6fb7ca846df819e9a930eff7e5f; no voters:
I20260808 15:49:14.367220 7295 raft_consensus.cc:2781] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:14.367745 7295 raft_consensus.cc:686] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 LEADER]: Becoming Leader. State: Replica: ee5ec6fb7ca846df819e9a930eff7e5f, State: Running, Role: LEADER
I20260808 15:49:14.368222 7295 consensus_queue.cc:237] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } }
I20260808 15:49:14.368899 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.368429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:40965)
0808 15:49:14.368635 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.368759 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:49:14.372354 6449 catalog_manager.cc:5495] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f reported cstate change: term changed from 0 to 1, leader changed from <none> to ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193). New cstate: current_term: 1 leader_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee5ec6fb7ca846df819e9a930eff7e5f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 33515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 } health_report { overall_health: UNKNOWN } } }
I20260808 15:49:14.412935 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.407282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.27.84.193:33515)
0808 15:49:14.407500 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.407504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.407526 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:49:14.407706 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.407710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.407843 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.407984 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:14.408379 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.409271 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.410669 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.411136 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.411261 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.411683 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:14.412337 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:14.412454 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.412588 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.412669 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:49:14.417714 6868 raft_consensus.cc:1261] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Refusing update from remote peer ee5ec6fb7ca846df819e9a930eff7e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:14.418002 7081 raft_consensus.cc:1261] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Refusing update from remote peer ee5ec6fb7ca846df819e9a930eff7e5f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:14.418701 7307 consensus_queue.cc:1035] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4b1753305774f8e81ce34ae635a4df3" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:14.419235 7295 consensus_queue.cc:1035] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb92e1212952499aaed5d509a19d8683" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:14.422674 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.421544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:46921)
0808 15:49:14.421794 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.422351 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260808 15:49:14.422940 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.422383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:60089)
0808 15:49:14.422580 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.422733 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260808 15:49:14.427171 7312 mvcc.cc:205] Tried to move back new op lower bound from 7316292216486559744 to 7316292216297025536. Current Snapshot: MvccSnapshot[applied={T|T < 7316292216486559744}]
I20260808 15:49:14.429744 7298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.427035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.194:44477)
0808 15:49:14.429345 (+ 2310us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.429493 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
I20260808 15:49:14.436355 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.428227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:35597)
0808 15:49:14.435865 (+ 7638us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.436049 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7438}
I20260808 15:49:14.439795 6449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:14.440104 6449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:14.449259 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.447678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.193:38617)
0808 15:49:14.448921 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.449051 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1118}
I20260808 15:49:14.451771 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.451192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.27.84.195:48141)
0808 15:49:14.451504 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.451598 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260808 15:49:14.455355 6449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f83ced808e574044a6d122cfe7d700ab: Sending DeleteTablet for 3 replicas of tablet 47af19bdfc61495db72b9a7bc5357346
I20260808 15:49:14.456590 7053 tablet_service.cc:1514] Processing DeleteTablet for tablet 47af19bdfc61495db72b9a7bc5357346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:14 UTC) from {username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:33628
I20260808 15:49:14.457654 6832 tablet_service.cc:1514] Processing DeleteTablet for tablet 47af19bdfc61495db72b9a7bc5357346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:14 UTC) from {username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:43602
I20260808 15:49:14.457816 6624 tablet_service.cc:1514] Processing DeleteTablet for tablet 47af19bdfc61495db72b9a7bc5357346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:14 UTC) from {username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:57716
I20260808 15:49:14.458273 6623 tablet_service.cc:1514] Processing DeleteTablet for tablet 47af19bdfc61495db72b9a7bc5357346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:14 UTC) from {username='oryx', principal='oryx/127.27.84.252@KRBTEST.COM'} at 127.0.0.1:57716
I20260808 15:49:14.460242 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 6403
I20260808 15:49:14.475050 7330 tablet_replica.cc:330] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: stopping tablet replica
I20260808 15:49:14.475340 7330 raft_consensus.cc:2229] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:14.476164 7330 raft_consensus.cc:2258] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:14.479081 7330 ts_tablet_manager.cc:1905] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:14.490451 7333 tablet_replica.cc:330] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: stopping tablet replica
I20260808 15:49:14.490736 7333 raft_consensus.cc:2229] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:14.491014 7330 ts_tablet_manager.cc:1918] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:14.491169 7330 log.cc:1196] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/47af19bdfc61495db72b9a7bc5357346
I20260808 15:49:14.491488 7333 raft_consensus.cc:2258] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:14.491732 7330 ts_tablet_manager.cc:1939] T 47af19bdfc61495db72b9a7bc5357346 P d4b1753305774f8e81ce34ae635a4df3: Deleting consensus metadata
I20260808 15:49:14.493287 7327 tablet_replica.cc:330] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: stopping tablet replica
I20260808 15:49:14.493552 7327 raft_consensus.cc:2229] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:14.493695 6424 catalog_manager.cc:4841] TS d4b1753305774f8e81ce34ae635a4df3 (127.27.84.194:40679): tablet 47af19bdfc61495db72b9a7bc5357346 (table test-table [id=679d654faeea4b6ba2aa0d73c87cb520]) successfully deleted
I20260808 15:49:14.493906 7327 raft_consensus.cc:2258] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:14.495134 7327 ts_tablet_manager.cc:1905] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:14.495934 7333 ts_tablet_manager.cc:1905] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:14.502149 7327 ts_tablet_manager.cc:1918] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:14.502336 7327 log.cc:1196] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/47af19bdfc61495db72b9a7bc5357346
I20260808 15:49:14.502868 7327 ts_tablet_manager.cc:1939] T 47af19bdfc61495db72b9a7bc5357346 P cb92e1212952499aaed5d509a19d8683: Deleting consensus metadata
I20260808 15:49:14.504197 6428 catalog_manager.cc:4841] TS cb92e1212952499aaed5d509a19d8683 (127.27.84.195:45919): tablet 47af19bdfc61495db72b9a7bc5357346 (table test-table [id=679d654faeea4b6ba2aa0d73c87cb520]) successfully deleted
I20260808 15:49:14.507061 7333 ts_tablet_manager.cc:1918] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:14.507308 7333 log.cc:1196] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/47af19bdfc61495db72b9a7bc5357346
I20260808 15:49:14.507896 7333 ts_tablet_manager.cc:1939] T 47af19bdfc61495db72b9a7bc5357346 P ee5ec6fb7ca846df819e9a930eff7e5f: Deleting consensus metadata
I20260808 15:49:14.509258 6426 catalog_manager.cc:4841] TS ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515): tablet 47af19bdfc61495db72b9a7bc5357346 (table test-table [id=679d654faeea4b6ba2aa0d73c87cb520]) successfully deleted
W20260808 15:49:14.509990 6426 catalog_manager.cc:4808] TS ee5ec6fb7ca846df819e9a930eff7e5f (127.27.84.193:33515): delete failed for tablet 47af19bdfc61495db72b9a7bc5357346 because the tablet was not found. No further retry: Not found: Tablet not found: 47af19bdfc61495db72b9a7bc5357346
I20260808 15:49:14.515693 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.462069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.27.84.193:33515)
0808 15:49:14.462744 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.462750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.492553 (+ 29803us) server_negotiation.cc:407] Connection header received
0808 15:49:14.492816 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.492823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.492910 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.493028 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:14.496078 (+ 3050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.497330 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.504042 (+ 6712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.504827 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.504980 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.508878 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:14.508901 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:14.508926 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:14.509000 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:14.512659 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:14.513443 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:14.513451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:14.513453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:14.513604 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:14.514035 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:14.514039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:14.514041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:14.514434 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:14.514832 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.515094 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.515334 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518}
I20260808 15:49:14.516407 6708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:14.634811 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 6403
I20260808 15:49:14.666045 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 6732
I20260808 15:49:14.680563 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.667022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.27.84.194:40679)
0808 15:49:14.667248 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.667255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.667276 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:49:14.667403 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.667409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.667487 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.667620 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:14.669006 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.669770 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.671159 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.671567 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.671708 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.674326 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:14.674343 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:14.674361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:14.674394 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:14.677548 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:14.678346 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:14.678354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:14.678356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:14.678490 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:14.678939 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:14.678944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:14.678948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:14.679368 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:14.679675 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.679970 (+ 295us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.680212 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:49:14.681365 6926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:14.779111 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 6732
I20260808 15:49:14.808058 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 6955
I20260808 15:49:14.821893 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.808923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.27.84.195:45919)
0808 15:49:14.809129 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:14.809136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:14.809263 (+ 127us) server_negotiation.cc:407] Connection header received
0808 15:49:14.809489 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:14.809494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:14.809583 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:14.809697 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:14.811209 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.811940 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.813171 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:14.813553 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:14.813647 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:14.816229 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:14.816246 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:14.816263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:14.816296 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:14.819286 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:14.819914 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:14.819921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:14.819924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:14.820049 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:14.820390 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:14.820393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:14.820395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:14.820748 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:14.821069 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:14.821428 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 15:49:14.821597 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:49:14.822656 7138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:14.869266 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.868453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:58006)
0808 15:49:14.868983 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.869080 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":294,"threads_started":1}
I20260808 15:49:14.870775 7339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:14.870318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39485 (local address 127.0.0.1:58018)
0808 15:49:14.870474 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:14.870557 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:39485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 15:49:14.946584 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 6955
I20260808 15:49:14.976760 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 5952
I20260808 15:49:14.978091 6206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:15.100060 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 5952
I20260808 15:49:15.129983 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 6107
I20260808 15:49:15.131220 6492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:15.260170 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 6107
2026-08-08T15:49:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10630 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:15 dist-test-slave-twwt krb5kdc[7348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:15 dist-test-slave-twwt krb5kdc[7348](info): set up 2 sockets
Aug 08 15:49:15 dist-test-slave-twwt krb5kdc[7348](info): commencing operation
krb5kdc: starting...
W20260808 15:49:16.645028 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.008s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:16 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab.
Entry for principal oryx/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab.
I20260808 15:49:16.942754 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42387
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab
--principal=oryx/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:17.286909 7366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:17.287212 7366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:17.287371 7366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:17.294874 7366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:17.295001 7366 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260808 15:49:17.295065 7366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:17.295167 7366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:17.295239 7366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:17.295324 7366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:17.305601 7366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42387
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.254.keytab
--principal=oryx/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:17.307039 7366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:17.308476 7366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:17.320987 7372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:17.321020 7375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:17.321089 7378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:17.322014 7366 server_base.cc:1034] running on GCE node
I20260808 15:49:17.322885 7366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:17.324283 7366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:17.325510 7366 hybrid_clock.cc:648] HybridClock initialized: now 1786204157325448 us; error 30 us; skew 500 ppm
I20260808 15:49:17.325871 7366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:17 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204157, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:17.329157 7366 init.cc:377] Logged in from keytab as oryx/127.27.84.254@KRBTEST.COM (short username oryx)
I20260808 15:49:17.330902 7366 webserver.cc:460] Webserver started at http://127.27.84.254:46435/ using document root <none> and password file <none>
I20260808 15:49:17.331638 7366 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:17.331774 7366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:17.332118 7366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:17.335443 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "09142f0c985d49539e62db93b42a53c0"
format_stamp: "Formatted at 2026-08-08 15:49:17 on dist-test-slave-twwt"
server_key: "d59e74a02856bcf71ef9a943c513f54f"
server_key_iv: "c6be1115e43354baf2b9666fb1a5b643"
server_key_version: "encryptionkey@0"
I20260808 15:49:17.336313 7366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "09142f0c985d49539e62db93b42a53c0"
format_stamp: "Formatted at 2026-08-08 15:49:17 on dist-test-slave-twwt"
server_key: "d59e74a02856bcf71ef9a943c513f54f"
server_key_iv: "c6be1115e43354baf2b9666fb1a5b643"
server_key_version: "encryptionkey@0"
I20260808 15:49:17.341862 7366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 15:49:17.346204 7386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:17.347862 7366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:49:17.348054 7366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "09142f0c985d49539e62db93b42a53c0"
format_stamp: "Formatted at 2026-08-08 15:49:17 on dist-test-slave-twwt"
server_key: "d59e74a02856bcf71ef9a943c513f54f"
server_key_iv: "c6be1115e43354baf2b9666fb1a5b643"
server_key_version: "encryptionkey@0"
I20260808 15:49:17.348281 7366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:17.361999 7366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:17.366828 7366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:17.367141 7366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:17.390558 7366 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:42387
I20260808 15:49:17.390578 7477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:42387 every 8 connection(s)
I20260808 15:49:17.392195 7366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:17.396235 7478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:17.401774 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 7366
I20260808 15:49:17.401995 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:17.402848 27987 external_mini_cluster.cc:1428] Setting key ffb45e8a027c96dd34d38369ef39df65
I20260808 15:49:17.408659 7478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Bootstrap starting.
I20260808 15:49:17.411934 7478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:17.413008 7478 log.cc:824] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:17.416745 7478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: No bootstrap required, opened a new log
Aug 08 15:49:17 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:17.427799 7478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09142f0c985d49539e62db93b42a53c0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42387 } }
I20260808 15:49:17.428299 7478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:17.428440 7478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09142f0c985d49539e62db93b42a53c0, State: Initialized, Role: FOLLOWER
I20260808 15:49:17.429044 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:17.405239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.27.84.254:42387)
0808 15:49:17.406232 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:17.406247 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:17.406292 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:49:17.408111 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:17.408135 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:17.408553 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:17.408970 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:17.410528 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:17.411897 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:17.413185 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:17.414300 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:17.414493 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:17.419497 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:17.419509 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:17.419535 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:17.419585 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:17.423883 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:17.424739 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:17.424749 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:17.424759 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:17.425008 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:17.425435 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:17.425440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:17.425444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:17.426283 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:17.426634 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:17.427114 (+ 480us) server_negotiation.cc:299] Negotiation successful
0808 15:49:17.427602 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":500,"threads_started":1}
I20260808 15:49:17.429155 7478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09142f0c985d49539e62db93b42a53c0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42387 } }
I20260808 15:49:17.429461 7478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:17.429652 7478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:17.429857 7478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:17.432336 7478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09142f0c985d49539e62db93b42a53c0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42387 } }
I20260808 15:49:17.432873 7478 leader_election.cc:302] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09142f0c985d49539e62db93b42a53c0; no voters:
I20260808 15:49:17.433494 7478 leader_election.cc:288] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:17.433645 7486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:17.434358 7486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [term 1 LEADER]: Becoming Leader. State: Replica: 09142f0c985d49539e62db93b42a53c0, State: Running, Role: LEADER
I20260808 15:49:17.434903 7486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09142f0c985d49539e62db93b42a53c0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42387 } }
I20260808 15:49:17.435662 7478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:17.437855 7490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09142f0c985d49539e62db93b42a53c0. Latest consensus state: current_term: 1 leader_uuid: "09142f0c985d49539e62db93b42a53c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09142f0c985d49539e62db93b42a53c0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42387 } } }
I20260808 15:49:17.438248 7490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:17.443423 7489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09142f0c985d49539e62db93b42a53c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09142f0c985d49539e62db93b42a53c0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42387 } } }
I20260808 15:49:17.443802 7489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:17.446256 7495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:17.451702 7495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:17.461720 7495 catalog_manager.cc:1347] Generated new cluster ID: 7cfb4c513e484d828bc73888b9f692dc
I20260808 15:49:17.461891 7495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:17.480391 7495 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:17.482295 7495 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:17.491793 7495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Generated new TSK 0
I20260808 15:49:17.492972 7495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab.
Entry for principal oryx/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab.
I20260808 15:49:17.563411 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42387
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:17.924896 7517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:17.925244 7517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:17.925421 7517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:17.933197 7517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:17.933373 7517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:17.933517 7517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:49:17.933660 7517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:17.944164 7517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:17.945732 7517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:17.947283 7517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:17.964529 7528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:17.964668 7529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:17.964749 7531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:17.965541 7517 server_base.cc:1034] running on GCE node
I20260808 15:49:17.966184 7517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:17.967306 7517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:17.968603 7517 hybrid_clock.cc:648] HybridClock initialized: now 1786204157968530 us; error 44 us; skew 500 ppm
I20260808 15:49:17.968997 7517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:17 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204157, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:17.977993 7517 init.cc:377] Logged in from keytab as oryx/127.27.84.193@KRBTEST.COM (short username oryx)
I20260808 15:49:17.979900 7517 webserver.cc:460] Webserver started at http://127.27.84.193:42735/ using document root <none> and password file <none>
I20260808 15:49:17.980727 7517 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:17.980891 7517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:17.981273 7517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:17.984630 7517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "46e8a2ef7db640af8712eb1fcd33445c"
format_stamp: "Formatted at 2026-08-08 15:49:17 on dist-test-slave-twwt"
server_key: "87e54b25a2593637e19f226f2db6298c"
server_key_iv: "99a7f07ef05bf1c88908ddf644e20651"
server_key_version: "encryptionkey@0"
I20260808 15:49:17.985558 7517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "46e8a2ef7db640af8712eb1fcd33445c"
format_stamp: "Formatted at 2026-08-08 15:49:17 on dist-test-slave-twwt"
server_key: "87e54b25a2593637e19f226f2db6298c"
server_key_iv: "99a7f07ef05bf1c88908ddf644e20651"
server_key_version: "encryptionkey@0"
I20260808 15:49:17.991495 7517 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260808 15:49:17.995580 7541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:17.997156 7517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260808 15:49:17.997344 7517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "46e8a2ef7db640af8712eb1fcd33445c"
format_stamp: "Formatted at 2026-08-08 15:49:17 on dist-test-slave-twwt"
server_key: "87e54b25a2593637e19f226f2db6298c"
server_key_iv: "99a7f07ef05bf1c88908ddf644e20651"
server_key_version: "encryptionkey@0"
I20260808 15:49:17.997678 7517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:18.011291 7517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:18.016115 7517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:18.016454 7517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:18.019824 7517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:18.019981 7517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:18.020144 7517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:18.020226 7517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:49:18 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204157, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:18.047533 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.020202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:50227 (local address 127.27.84.254:42387)
0808 15:49:18.020729 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:18.020736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:18.022110 (+ 1374us) server_negotiation.cc:407] Connection header received
0808 15:49:18.024324 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:18.024330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:18.024435 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:18.024563 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:18.026824 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.027947 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.030732 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.031385 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.031552 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.037688 (+ 6136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:18.037693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:18.037697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:18.037735 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:18.041242 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.042500 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.042506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.042509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.042716 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.043249 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.043254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.043257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.043563 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:18.043729 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:18.046804 (+ 3075us) server_negotiation.cc:299] Negotiation successful
0808 15:49:18.047089 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
I20260808 15:49:18.048237 7556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.020918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.27.84.193:50227)
0808 15:49:18.021886 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:18.021935 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:18.024005 (+ 2070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:18.024841 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:18.024850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:18.025525 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:18.026554 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.026588 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.028160 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.028172 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.030516 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.030540 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.031702 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.031710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.031963 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.032870 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:18.032921 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:18.037414 (+ 4493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:18.041473 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.041484 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.041503 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.042349 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.042851 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.042857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.042860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.043106 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.043720 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:18.043729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:18.044207 (+ 478us) client_negotiation.cc:769] Sending connection context
0808 15:49:18.044617 (+ 410us) client_negotiation.cc:240] Negotiation successful
0808 15:49:18.045246 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":403,"threads_started":1}
I20260808 15:49:18.067405 7517 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:38349
I20260808 15:49:18.067469 7725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:38349 every 8 connection(s)
I20260808 15:49:18.069203 7517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:18.071861 7726 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42387
I20260808 15:49:18.072124 7726 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:18.072877 7726 heartbeater.cc:511] Master 127.27.84.254:42387 requested a full tablet report, sending...
I20260808 15:49:18.073567 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.056483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:57661 (local address 127.27.84.254:42387)
0808 15:49:18.056686 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:18.056692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:18.057404 (+ 712us) server_negotiation.cc:407] Connection header received
0808 15:49:18.057655 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:18.057664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:18.057773 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:18.057884 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:18.059500 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.060766 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.063595 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.064343 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.064473 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.066602 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:18.066650 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:18.066656 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:18.066702 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:18.070375 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.071355 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.071361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.071364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.071527 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.071975 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.072005 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.072010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.072371 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:18.072527 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:18.072984 (+ 457us) server_negotiation.cc:299] Negotiation successful
0808 15:49:18.073205 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:49:18.073702 7687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.056537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.27.84.193:57661)
0808 15:49:18.057266 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:18.057293 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:18.057477 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:18.057949 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:18.057955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:18.058845 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:18.059332 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.059344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.060939 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.060949 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.063412 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.063433 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.064569 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.064577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.064733 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.065573 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:18.065591 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:18.066383 (+ 792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:18.070515 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.070521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.070525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.071225 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.071699 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.071705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.071709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.071853 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.072498 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:18.072504 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:18.072718 (+ 214us) client_negotiation.cc:769] Sending connection context
0808 15:49:18.072870 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 15:49:18.073205 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":396,"threads_started":1}
I20260808 15:49:18.075312 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 7517
I20260808 15:49:18.075560 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:18.075600 7410 ts_manager.cc:194] Registered new tserver with Master: 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349)
I20260808 15:49:18.076460 27987 external_mini_cluster.cc:1428] Setting key adcf610f88731c1dcbb50845079c03a6
I20260808 15:49:18.077275 7410 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:50227
WARNING: no policy specified for oryx/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab.
Entry for principal oryx/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab.
I20260808 15:49:18.140156 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42387
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:18.496538 7732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:18.496919 7732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:18.497113 7732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:18.504667 7732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:18.504823 7732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:18.504968 7732 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:49:18.505101 7732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:18.515261 7732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:18.516670 7732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:18.518221 7732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:18.532160 7738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:18.532116 7739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:18.532205 7744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:18.533058 7732 server_base.cc:1034] running on GCE node
I20260808 15:49:18.533655 7732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:18.534667 7732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:18.535917 7732 hybrid_clock.cc:648] HybridClock initialized: now 1786204158535825 us; error 66 us; skew 500 ppm
I20260808 15:49:18.536309 7732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:18 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204158, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:18.540086 7732 init.cc:377] Logged in from keytab as oryx/127.27.84.194@KRBTEST.COM (short username oryx)
I20260808 15:49:18.541622 7732 webserver.cc:460] Webserver started at http://127.27.84.194:40023/ using document root <none> and password file <none>
I20260808 15:49:18.542363 7732 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:18.542459 7732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:18.542847 7732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:18.545874 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "90189dc1cb624f0ab23ac22ad88061ac"
format_stamp: "Formatted at 2026-08-08 15:49:18 on dist-test-slave-twwt"
server_key: "c58cedb684459eb2db457fe9b842971d"
server_key_iv: "2c474f48f2b1cd8382344b69802ee0b2"
server_key_version: "encryptionkey@0"
I20260808 15:49:18.546737 7732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "90189dc1cb624f0ab23ac22ad88061ac"
format_stamp: "Formatted at 2026-08-08 15:49:18 on dist-test-slave-twwt"
server_key: "c58cedb684459eb2db457fe9b842971d"
server_key_iv: "2c474f48f2b1cd8382344b69802ee0b2"
server_key_version: "encryptionkey@0"
I20260808 15:49:18.552870 7732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 15:49:18.556658 7756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:18.558211 7732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:18.558434 7732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "90189dc1cb624f0ab23ac22ad88061ac"
format_stamp: "Formatted at 2026-08-08 15:49:18 on dist-test-slave-twwt"
server_key: "c58cedb684459eb2db457fe9b842971d"
server_key_iv: "2c474f48f2b1cd8382344b69802ee0b2"
server_key_version: "encryptionkey@0"
I20260808 15:49:18.558732 7732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:18.586779 7732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:18.591502 7732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:18.591769 7732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:18.594798 7732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:18.594921 7732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:18.595063 7732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:18.595146 7732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:49:18 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204158, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:18.621914 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.595730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36151 (local address 127.27.84.254:42387)
0808 15:49:18.596245 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:18.596254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:18.598026 (+ 1772us) server_negotiation.cc:407] Connection header received
0808 15:49:18.600752 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:18.600762 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:18.600853 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:18.600923 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:18.603903 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.605317 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.607634 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.608269 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.608437 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.613990 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:18.613996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:18.613999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:18.614035 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:18.617926 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.619059 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.619068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.619074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.619277 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.619878 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.619881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.619884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.620172 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:18.620307 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:18.621439 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0808 15:49:18.621619 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260808 15:49:18.624616 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.596578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.27.84.194:36151)
0808 15:49:18.597791 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:18.597848 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:18.600314 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:18.601622 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:18.601633 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:18.602720 (+ 1087us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:18.603668 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.603705 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.605529 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.605536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.607466 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.607479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.608571 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.608583 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.608956 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.610044 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:18.610084 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:18.613745 (+ 3661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:18.618123 (+ 4378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.618137 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.618163 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.618895 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.619427 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.619431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.619433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.619665 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.620330 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:18.620351 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:18.620951 (+ 600us) client_negotiation.cc:769] Sending connection context
0808 15:49:18.621356 (+ 405us) client_negotiation.cc:240] Negotiation successful
0808 15:49:18.622511 (+ 1155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":461,"threads_started":1}
I20260808 15:49:18.647634 7732 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:42303
I20260808 15:49:18.647704 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.631737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:42441 (local address 127.27.84.254:42387)
0808 15:49:18.631916 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:18.631920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:18.632727 (+ 807us) server_negotiation.cc:407] Connection header received
0808 15:49:18.632954 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:18.632959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:18.633036 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:18.633141 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:18.634134 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.634944 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.636440 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.637112 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.637255 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.640154 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:18.640161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:18.640166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:18.640211 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:18.644254 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.645350 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.645354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.645356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.645477 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:18.645905 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:18.645909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:18.645911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:18.646227 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:18.646391 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:18.647008 (+ 617us) server_negotiation.cc:299] Negotiation successful
0808 15:49:18.647292 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:49:18.647975 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:18.631916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.27.84.194:42441)
0808 15:49:18.632589 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:18.632606 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:18.632811 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:18.633173 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:18.633178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:18.633686 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:18.633985 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.633994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.635105 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.635109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.636300 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:18.636312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:18.637385 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:18.637395 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:18.637664 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:18.638500 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:18.638521 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:18.639855 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:18.644454 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.644462 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.644468 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.645225 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.645591 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:18.645596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:18.645599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:18.645792 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:18.646364 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:18.646369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:18.646689 (+ 320us) client_negotiation.cc:769] Sending connection context
0808 15:49:18.646887 (+ 198us) client_negotiation.cc:240] Negotiation successful
0808 15:49:18.647358 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":388,"threads_started":1}
I20260808 15:49:18.648722 7944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:42303 every 8 connection(s)
I20260808 15:49:18.649298 7732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:18.650761 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 7732
I20260808 15:49:18.650952 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:18.651608 27987 external_mini_cluster.cc:1428] Setting key efa6c79cae6fb498f16f55c39268bd37
I20260808 15:49:18.653432 7945 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42387
I20260808 15:49:18.653697 7945 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:18.654547 7945 heartbeater.cc:511] Master 127.27.84.254:42387 requested a full tablet report, sending...
I20260808 15:49:18.656046 7410 ts_manager.cc:194] Registered new tserver with Master: 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303)
I20260808 15:49:18.656857 7410 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:36151
WARNING: no policy specified for oryx/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab.
Entry for principal oryx/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab.
I20260808 15:49:18.714745 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42387
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:19.074519 7954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:19.074903 7954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:19.075098 7954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:19.082484 7954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:19.082671 7954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:19.082824 7954 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260808 15:49:19.082964 7954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:19.085318 7726 heartbeater.cc:503] Master 127.27.84.254:42387 was elected leader, sending a full tablet report...
I20260808 15:49:19.093341 7954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.84.254:42387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:19.094853 7954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:19.096283 7954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:19.110046 7959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:19.110109 7965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:19.110157 7961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:19.111467 7954 server_base.cc:1034] running on GCE node
I20260808 15:49:19.112177 7954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:19.113211 7954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:19.114434 7954 hybrid_clock.cc:648] HybridClock initialized: now 1786204159114345 us; error 61 us; skew 500 ppm
I20260808 15:49:19.114792 7954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:19 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204159, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:19.118134 7954 init.cc:377] Logged in from keytab as oryx/127.27.84.195@KRBTEST.COM (short username oryx)
I20260808 15:49:19.120009 7954 webserver.cc:460] Webserver started at http://127.27.84.195:37257/ using document root <none> and password file <none>
I20260808 15:49:19.120774 7954 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:19.120898 7954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:19.121264 7954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:19.124540 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
format_stamp: "Formatted at 2026-08-08 15:49:19 on dist-test-slave-twwt"
server_key: "aeadc4f205ee7038a4fa12dfe67a5918"
server_key_iv: "b4383c9eb38d4b16ed9974bb86f8699a"
server_key_version: "encryptionkey@0"
I20260808 15:49:19.125590 7954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
format_stamp: "Formatted at 2026-08-08 15:49:19 on dist-test-slave-twwt"
server_key: "aeadc4f205ee7038a4fa12dfe67a5918"
server_key_iv: "b4383c9eb38d4b16ed9974bb86f8699a"
server_key_version: "encryptionkey@0"
I20260808 15:49:19.131315 7954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:49:19.135531 7980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:19.137384 7954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:49:19.137591 7954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
format_stamp: "Formatted at 2026-08-08 15:49:19 on dist-test-slave-twwt"
server_key: "aeadc4f205ee7038a4fa12dfe67a5918"
server_key_iv: "b4383c9eb38d4b16ed9974bb86f8699a"
server_key_version: "encryptionkey@0"
I20260808 15:49:19.137866 7954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:19.161130 7954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:19.166157 7954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:19.166460 7954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:19.169404 7954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:19.169492 7954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:19.169654 7954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:19.169735 7954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:49:19 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204159, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:19.195468 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:19.170414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:40153 (local address 127.27.84.254:42387)
0808 15:49:19.170929 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:19.170934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:19.172454 (+ 1520us) server_negotiation.cc:407] Connection header received
0808 15:49:19.174868 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:19.174878 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:19.175002 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:19.175143 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:19.177742 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.178905 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.181591 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.182109 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.182264 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:19.188085 (+ 5821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:19.188091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:19.188094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:19.188131 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:19.191893 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:19.192632 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:19.192640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:19.192645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:19.192802 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:19.193259 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:19.193265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:19.193268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:19.193544 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:19.193704 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:19.194530 (+ 826us) server_negotiation.cc:299] Negotiation successful
0808 15:49:19.194853 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":276,"threads_started":1}
I20260808 15:49:19.196463 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:19.171296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.27.84.195:40153)
0808 15:49:19.172266 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:19.172317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:19.174412 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:19.175451 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:19.175460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:19.176281 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:19.177536 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:19.177560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.179102 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.179110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:19.181423 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:19.181444 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.182432 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.182448 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:19.182887 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:19.183912 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:19.183963 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:19.187784 (+ 3821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:19.192043 (+ 4259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:19.192052 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:19.192073 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:19.192506 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:19.192918 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:19.192923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:19.192927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:19.193141 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:19.193679 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:19.193688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:19.194059 (+ 371us) client_negotiation.cc:769] Sending connection context
0808 15:49:19.194370 (+ 311us) client_negotiation.cc:240] Negotiation successful
0808 15:49:19.194987 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":426,"threads_started":1}
I20260808 15:49:19.220870 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:19.203181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:40717 (local address 127.27.84.254:42387)
0808 15:49:19.203381 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:19.203387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:19.204372 (+ 985us) server_negotiation.cc:407] Connection header received
0808 15:49:19.204599 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:19.204603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:19.204665 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:19.204738 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:19.206266 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.207124 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.209873 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.210446 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.210648 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:19.213452 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:19.213458 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:19.213461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:19.213497 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:19.217073 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:19.218211 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:19.218218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:19.218224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:19.218403 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:19.219169 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:19.219175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:19.219179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:19.219596 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:19.219779 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:19.220382 (+ 603us) server_negotiation.cc:299] Negotiation successful
0808 15:49:19.220570 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:49:19.221460 8097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:19.203341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.27.84.195:40717)
0808 15:49:19.204230 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:19.204256 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:19.204446 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:19.204809 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:19.204817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:19.205614 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:19.206098 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:19.206116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.207292 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.207300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:19.209611 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:19.209632 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.210723 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.210733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:19.210976 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:19.211755 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:19.211788 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:19.213202 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:19.217232 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:19.217240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:19.217252 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:19.218049 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:19.218547 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:19.218553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:19.218557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:19.218986 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:19.219773 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:19.219782 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:19.220068 (+ 286us) client_negotiation.cc:769] Sending connection context
0808 15:49:19.220274 (+ 206us) client_negotiation.cc:240] Negotiation successful
0808 15:49:19.220763 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":529,"threads_started":1}
I20260808 15:49:19.228951 7954 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:39133
I20260808 15:49:19.228998 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:39133 every 8 connection(s)
I20260808 15:49:19.230154 7954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:19.232236 8170 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42387
I20260808 15:49:19.232488 8170 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:19.233150 8170 heartbeater.cc:511] Master 127.27.84.254:42387 requested a full tablet report, sending...
I20260808 15:49:19.234468 7410 ts_manager.cc:194] Registered new tserver with Master: 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:19.235330 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 7954
I20260808 15:49:19.235410 7410 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:40153
I20260808 15:49:19.235513 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:19.236135 27987 external_mini_cluster.cc:1428] Setting key 8487eed82fc45a128ed038f5cc507332
I20260808 15:49:19.239147 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 08 15:49:19 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 08 15:49:19 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:19 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:19.660118 7945 heartbeater.cc:503] Master 127.27.84.254:42387 was elected leader, sending a full tablet report...
I20260808 15:49:20.238389 8170 heartbeater.cc:503] Master 127.27.84.254:42387 was elected leader, sending a full tablet report...
W20260808 15:49:20.443213 8183 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.143s user 0.003s sys 0.019s
W20260808 15:49:20.444630 7995 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:19.291700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.27.84.254:42387)
0808 15:49:19.291943 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:19.291950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:19.292285 (+ 335us) server_negotiation.cc:407] Connection header received
0808 15:49:19.292556 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:19.292564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:19.292666 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:19.292801 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:19.295079 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.296416 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.297833 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:19.298601 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:19.298781 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.444291 (+1145510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33276: BlockingRecv error: recv got EOF from 127.0.0.1:33276 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 15:49:20 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204159, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:20.466521 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.450055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.27.84.254:42387)
0808 15:49:20.450364 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.450372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.450864 (+ 492us) server_negotiation.cc:407] Connection header received
0808 15:49:20.451111 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.451117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.451210 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.451332 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:20.452909 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.453760 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.455279 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.455773 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.455930 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.460501 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:20.460508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:20.460511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:20.460550 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:20.463773 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:20.464561 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:20.464566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:20.464568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:20.464711 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:20.465209 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:20.465214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:20.465216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:20.465517 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:20.465713 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.466085 (+ 372us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.466282 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 15:49:20.471038 7410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:20.478071 7410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:20.513458 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.492729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.27.84.194:42303)
0808 15:49:20.504370 (+ 11641us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.504381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.505021 (+ 640us) server_negotiation.cc:407] Connection header received
0808 15:49:20.505173 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.505186 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.505529 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.506750 (+ 1221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.507023 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.508901 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.511276 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.512246 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.512410 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.512645 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.512975 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.513144 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11444,"thread_start_us":11281,"threads_started":1}
I20260808 15:49:20.513525 8204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.492037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42303 (local address 127.0.0.1:44448)
0808 15:49:20.504723 (+ 12686us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.504753 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.504880 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.505851 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.505862 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.505923 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.506462 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.506478 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.509049 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.509054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.511103 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.511117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.512484 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.512491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.512733 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.512772 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.512918 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.513113 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12561,"mutex_wait_us":20,"thread_start_us":12410,"threads_started":1}
I20260808 15:49:20.515560 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.504903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45060 (local address 127.27.84.195:39133)
0808 15:49:20.507501 (+ 2598us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.507510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.507653 (+ 143us) server_negotiation.cc:407] Connection header received
0808 15:49:20.507770 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.507778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.507989 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.508158 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.508967 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.509930 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.513112 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.514534 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.514710 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.514924 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.515118 (+ 194us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.515264 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2444,"thread_start_us":2304,"threads_started":1}
I20260808 15:49:20.515612 8206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.504641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:39133 (local address 127.0.0.1:45060)
0808 15:49:20.506894 (+ 2253us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.506910 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.507011 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.508363 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.508368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.508394 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.508767 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.508781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.510087 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.510092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.512917 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.512937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.514758 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.514765 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.514916 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.514953 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.515069 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.515192 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2145,"mutex_wait_us":189,"thread_start_us":2578,"threads_started":1}
I20260808 15:49:20.516311 8203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.491541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:38349 (local address 127.0.0.1:49586)
0808 15:49:20.504130 (+ 12589us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.504179 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.504377 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.507469 (+ 3092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.507473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.507499 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.507939 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.507954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.510099 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.510106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.512944 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.512954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.515503 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.515510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.515700 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.515733 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.515807 (+ 74us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.515943 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12322,"mutex_wait_us":198,"thread_start_us":12175,"threads_started":1}
I20260808 15:49:20.517246 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.491670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.27.84.193:38349)
0808 15:49:20.506624 (+ 14954us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.506635 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.506668 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:49:20.506798 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.506806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.507074 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.508146 (+ 1072us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.508348 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.509941 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.513112 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.515247 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.516534 (+ 1287us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.516731 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.516799 (+ 68us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.516963 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14759,"thread_start_us":14611,"threads_started":1}
I20260808 15:49:20.518013 7838 tablet_service.cc:1467] Processing CreateTablet for tablet 80397c96d983465188a4c946182bede2 (DEFAULT_TABLE table=test-table [id=83f7ce8e7ba3415a83205c6d22c25b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:20.519629 7838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80397c96d983465188a4c946182bede2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.521347 8051 tablet_service.cc:1467] Processing CreateTablet for tablet 80397c96d983465188a4c946182bede2 (DEFAULT_TABLE table=test-table [id=83f7ce8e7ba3415a83205c6d22c25b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:20.522869 8051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80397c96d983465188a4c946182bede2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.524721 7623 tablet_service.cc:1467] Processing CreateTablet for tablet 80397c96d983465188a4c946182bede2 (DEFAULT_TABLE table=test-table [id=83f7ce8e7ba3415a83205c6d22c25b3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:20.525864 7623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80397c96d983465188a4c946182bede2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.535769 8210 tablet_bootstrap.cc:492] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Bootstrap starting.
I20260808 15:49:20.538115 8211 tablet_bootstrap.cc:492] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Bootstrap starting.
I20260808 15:49:20.538971 8213 tablet_bootstrap.cc:492] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Bootstrap starting.
I20260808 15:49:20.541314 8210 tablet_bootstrap.cc:654] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.542527 8213 tablet_bootstrap.cc:654] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.542541 8211 tablet_bootstrap.cc:654] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.542614 8210 log.cc:824] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:20.543830 8211 log.cc:824] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:20.543830 8213 log.cc:824] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:20.545948 8210 tablet_bootstrap.cc:492] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: No bootstrap required, opened a new log
I20260808 15:49:20.546257 8210 ts_tablet_manager.cc:1397] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 15:49:20.547042 8213 tablet_bootstrap.cc:492] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: No bootstrap required, opened a new log
I20260808 15:49:20.547055 8211 tablet_bootstrap.cc:492] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: No bootstrap required, opened a new log
I20260808 15:49:20.547361 8213 ts_tablet_manager.cc:1397] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:49:20.547385 8211 ts_tablet_manager.cc:1397] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:49:20.555492 8213 raft_consensus.cc:348] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.555892 8213 raft_consensus.cc:374] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.555994 8213 raft_consensus.cc:729] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.556635 8213 consensus_queue.cc:260] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.557818 8210 raft_consensus.cc:348] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.558338 8210 raft_consensus.cc:374] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.558480 8210 raft_consensus.cc:729] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.558490 8213 ts_tablet_manager.cc:1428] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260808 15:49:20.558261 8211 raft_consensus.cc:348] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.558756 8211 raft_consensus.cc:374] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.558914 8211 raft_consensus.cc:729] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b4e52b4d0444aaa9c6b3b01813da6d, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.559243 8210 consensus_queue.cc:260] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.559499 8211 consensus_queue.cc:260] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.561378 8210 ts_tablet_manager.cc:1428] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260808 15:49:20.563342 8211 ts_tablet_manager.cc:1428] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent starting tablet: real 0.016s user 0.006s sys 0.009s
W20260808 15:49:20.573237 7727 tablet.cc:2369] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:20.599325 8220 raft_consensus.cc:482] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:20.599710 8220 raft_consensus.cc:504] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.601761 8220 leader_election.cc:288] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:20.610538 8201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.602275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:46065 (local address 127.27.84.193:38349)
0808 15:49:20.602579 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.602585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.602979 (+ 394us) server_negotiation.cc:407] Connection header received
0808 15:49:20.603355 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.603364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.603471 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.603601 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.604393 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.605374 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.608340 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.609561 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.609741 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.609826 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.610169 (+ 343us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.610300 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 15:49:20.610882 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.602081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:38349 (local address 127.27.84.194:46065)
0808 15:49:20.602826 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.602844 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.603049 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.603651 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.603660 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.603706 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.604222 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.604235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.605534 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.605539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.608144 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.608165 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.609805 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.609810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.609945 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.609972 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.610121 (+ 149us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.610305 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":53,"thread_start_us":396,"threads_started":1}
I20260808 15:49:20.611186 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.603288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:51653 (local address 127.27.84.195:39133)
0808 15:49:20.603500 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.603509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.604144 (+ 635us) server_negotiation.cc:407] Connection header received
0808 15:49:20.604367 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.604371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.604451 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.604563 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.605095 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.605961 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.608682 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.609932 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.610054 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.610156 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.610558 (+ 402us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.610896 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:49:20.611217 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.602974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:39133 (local address 127.27.84.194:51653)
0808 15:49:20.604018 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.604042 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.604208 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.604617 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.604621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.604644 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.604933 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.604940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.606132 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.606138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.608502 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.608514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.610177 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.610184 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.610356 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.610390 (+ 34us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.610504 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.610675 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"mutex_wait_us":57,"thread_start_us":637,"threads_started":1}
I20260808 15:49:20.612176 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80397c96d983465188a4c946182bede2" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e8a2ef7db640af8712eb1fcd33445c" is_pre_election: true
I20260808 15:49:20.612355 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80397c96d983465188a4c946182bede2" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" is_pre_election: true
I20260808 15:49:20.612967 8089 raft_consensus.cc:2454] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 0.
I20260808 15:49:20.612967 7648 raft_consensus.cc:2454] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 0.
I20260808 15:49:20.613850 7758 leader_election.cc:302] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e8a2ef7db640af8712eb1fcd33445c, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:20.614347 8220 raft_consensus.cc:2781] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:20.614544 8220 raft_consensus.cc:482] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:20.614717 8220 raft_consensus.cc:3037] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:20.617308 8220 raft_consensus.cc:504] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.618129 8220 leader_election.cc:288] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 election: Requested vote from peers 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:20.618885 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80397c96d983465188a4c946182bede2" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
I20260808 15:49:20.618863 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80397c96d983465188a4c946182bede2" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e8a2ef7db640af8712eb1fcd33445c"
I20260808 15:49:20.619136 8089 raft_consensus.cc:3037] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:20.619191 7648 raft_consensus.cc:3037] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:20.621613 7648 raft_consensus.cc:2454] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 1.
I20260808 15:49:20.621752 8089 raft_consensus.cc:2454] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 1.
I20260808 15:49:20.622243 7758 leader_election.cc:302] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e8a2ef7db640af8712eb1fcd33445c, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:20.622714 8220 raft_consensus.cc:2781] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:20.623289 8220 raft_consensus.cc:686] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 LEADER]: Becoming Leader. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Running, Role: LEADER
I20260808 15:49:20.623862 8220 consensus_queue.cc:237] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } }
I20260808 15:49:20.629460 7412 catalog_manager.cc:5495] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194). New cstate: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } health_report { overall_health: HEALTHY } } }
W20260808 15:49:20.652662 7947 tablet.cc:2369] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:20.658097 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.652401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.27.84.194:42303)
0808 15:49:20.652641 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.652651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.652797 (+ 146us) server_negotiation.cc:407] Connection header received
0808 15:49:20.653065 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.653069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.653151 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.653319 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:20.653719 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.654514 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.655904 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.656333 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.656474 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.656936 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:20.657481 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:20.657659 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.657713 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.657798 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:49:20.667209 8089 raft_consensus.cc:1261] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Refusing update from remote peer 90189dc1cb624f0ab23ac22ad88061ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:20.667306 7648 raft_consensus.cc:1261] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Refusing update from remote peer 90189dc1cb624f0ab23ac22ad88061ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:20.668390 8224 consensus_queue.cc:1035] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:20.668843 8220 consensus_queue.cc:1035] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:20.682843 8229 mvcc.cc:205] Tried to move back new op lower bound from 7316292242081546240 to 7316292241919733760. Current Snapshot: MvccSnapshot[applied={T|T < 7316292242081546240}]
I20260808 15:49:20.684788 8230 mvcc.cc:205] Tried to move back new op lower bound from 7316292242081546240 to 7316292241919733760. Current Snapshot: MvccSnapshot[applied={T|T < 7316292242081546240}]
I20260808 15:49:20.698240 7412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:20.698570 7412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:20.702845 7412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Sending DeleteTablet for 3 replicas of tablet 80397c96d983465188a4c946182bede2
I20260808 15:49:20.704190 7838 tablet_service.cc:1514] Processing DeleteTablet for tablet 80397c96d983465188a4c946182bede2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:20 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:44448
I20260808 15:49:20.704288 7623 tablet_service.cc:1514] Processing DeleteTablet for tablet 80397c96d983465188a4c946182bede2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:20 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49586
I20260808 15:49:20.704288 8051 tablet_service.cc:1514] Processing DeleteTablet for tablet 80397c96d983465188a4c946182bede2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:20 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45060
I20260808 15:49:20.705907 7410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:20.706908 7410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:20.717614 7622 tablet_service.cc:1467] Processing CreateTablet for tablet 485a9918c78c4de5834d3ff1c6ed99aa (DEFAULT_TABLE table=test-table [id=5a66fcf9d63844f89dd04e01df4f621c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:20.718353 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485a9918c78c4de5834d3ff1c6ed99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.718230 7837 tablet_service.cc:1467] Processing CreateTablet for tablet 485a9918c78c4de5834d3ff1c6ed99aa (DEFAULT_TABLE table=test-table [id=5a66fcf9d63844f89dd04e01df4f621c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:20.719100 7837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485a9918c78c4de5834d3ff1c6ed99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.724438 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 485a9918c78c4de5834d3ff1c6ed99aa (DEFAULT_TABLE table=test-table [id=5a66fcf9d63844f89dd04e01df4f621c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:20.725018 8242 tablet_replica.cc:330] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: stopping tablet replica
I20260808 15:49:20.725072 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 485a9918c78c4de5834d3ff1c6ed99aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.728345 8242 raft_consensus.cc:2229] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:20.729008 8242 raft_consensus.cc:2258] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:20.729664 8240 tablet_replica.cc:330] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: stopping tablet replica
I20260808 15:49:20.730156 8210 tablet_bootstrap.cc:492] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Bootstrap starting.
I20260808 15:49:20.732542 8240 raft_consensus.cc:2229] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:20.733412 8240 raft_consensus.cc:2258] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:20.733731 8211 tablet_bootstrap.cc:492] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Bootstrap starting.
I20260808 15:49:20.734632 8210 tablet_bootstrap.cc:654] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.735522 8211 tablet_bootstrap.cc:654] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.737900 8241 tablet_replica.cc:330] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: stopping tablet replica
I20260808 15:49:20.739008 8210 tablet_bootstrap.cc:492] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: No bootstrap required, opened a new log
I20260808 15:49:20.739176 8210 ts_tablet_manager.cc:1397] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260808 15:49:20.739632 8211 tablet_bootstrap.cc:492] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: No bootstrap required, opened a new log
I20260808 15:49:20.739777 8211 ts_tablet_manager.cc:1397] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260808 15:49:20.740049 8242 ts_tablet_manager.cc:1905] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:20.739576 8241 raft_consensus.cc:2229] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:20.740049 8210 raft_consensus.cc:348] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.740306 8210 raft_consensus.cc:374] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.740417 8210 raft_consensus.cc:729] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.740635 8211 raft_consensus.cc:348] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.740739 8210 consensus_queue.cc:260] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.741472 8210 ts_tablet_manager.cc:1428] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260808 15:49:20.740833 8241 raft_consensus.cc:2258] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:20.740880 8211 raft_consensus.cc:374] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.742252 8211 raft_consensus.cc:729] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b4e52b4d0444aaa9c6b3b01813da6d, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.742729 8211 consensus_queue.cc:260] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.743319 8211 ts_tablet_manager.cc:1428] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:20.745023 8240 ts_tablet_manager.cc:1905] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:20.747380 8213 tablet_bootstrap.cc:492] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Bootstrap starting.
I20260808 15:49:20.748894 8241 ts_tablet_manager.cc:1905] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:20.749131 8213 tablet_bootstrap.cc:654] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.753654 8240 ts_tablet_manager.cc:1918] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:20.753854 8240 log.cc:1196] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/80397c96d983465188a4c946182bede2
I20260808 15:49:20.754451 8240 ts_tablet_manager.cc:1939] T 80397c96d983465188a4c946182bede2 P 90189dc1cb624f0ab23ac22ad88061ac: Deleting consensus metadata
I20260808 15:49:20.756213 8241 ts_tablet_manager.cc:1918] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:20.756428 8241 log.cc:1196] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/80397c96d983465188a4c946182bede2
I20260808 15:49:20.756376 7391 catalog_manager.cc:4841] TS 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303): tablet 80397c96d983465188a4c946182bede2 (table test-table [id=83f7ce8e7ba3415a83205c6d22c25b3c]) successfully deleted
I20260808 15:49:20.757009 8241 ts_tablet_manager.cc:1939] T 80397c96d983465188a4c946182bede2 P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting consensus metadata
I20260808 15:49:20.757062 8242 ts_tablet_manager.cc:1918] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:20.757256 8242 log.cc:1196] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/80397c96d983465188a4c946182bede2
I20260808 15:49:20.757895 8242 ts_tablet_manager.cc:1939] T 80397c96d983465188a4c946182bede2 P 46e8a2ef7db640af8712eb1fcd33445c: Deleting consensus metadata
I20260808 15:49:20.759017 7388 catalog_manager.cc:4841] TS 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133): tablet 80397c96d983465188a4c946182bede2 (table test-table [id=83f7ce8e7ba3415a83205c6d22c25b3c]) successfully deleted
I20260808 15:49:20.759716 7388 catalog_manager.cc:4841] TS 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349): tablet 80397c96d983465188a4c946182bede2 (table test-table [id=83f7ce8e7ba3415a83205c6d22c25b3c]) successfully deleted
I20260808 15:49:20.760799 8213 tablet_bootstrap.cc:492] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: No bootstrap required, opened a new log
I20260808 15:49:20.760974 8213 ts_tablet_manager.cc:1397] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260808 15:49:20.761901 8213 raft_consensus.cc:348] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.762213 8213 raft_consensus.cc:374] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.762337 8213 raft_consensus.cc:729] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.762722 8213 consensus_queue.cc:260] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.763991 8213 ts_tablet_manager.cc:1428] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260808 15:49:20.824436 8218 raft_consensus.cc:482] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:20.824782 8218 raft_consensus.cc:504] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.826473 8218 leader_election.cc:288] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:20.835289 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.827818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:48503 (local address 127.27.84.194:42303)
0808 15:49:20.828027 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.828034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.828626 (+ 592us) server_negotiation.cc:407] Connection header received
0808 15:49:20.828813 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.828819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.828899 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.829003 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.829647 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.830883 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.833247 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.834063 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.834288 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.834374 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.834779 (+ 405us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.834944 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:49:20.835415 8207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.827543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:50661 (local address 127.27.84.195:39133)
0808 15:49:20.827737 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.827743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.827766 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:49:20.828021 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.828047 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.828150 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.828231 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:20.829165 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.830180 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.832857 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.834152 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.834346 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.834428 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.834950 (+ 522us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.835104 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:49:20.835340 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.826914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42303 (local address 127.27.84.193:48503)
0808 15:49:20.828504 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.828521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.828661 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.829056 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.829063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.829092 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.829458 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.829470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.831039 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.831045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.833065 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.833075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.834309 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.834316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.834549 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.834585 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.834740 (+ 155us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.834904 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1438,"mutex_wait_us":1,"thread_start_us":1242,"threads_started":1}
I20260808 15:49:20.836265 8247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.826837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:39133 (local address 127.27.84.193:50661)
0808 15:49:20.827569 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.827592 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.827815 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.828503 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.828509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.828545 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:20.828948 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.828958 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.830337 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.830347 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.832640 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.832658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.834553 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.834559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.834765 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.834795 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.835624 (+ 829us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.835780 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":36,"thread_start_us":415,"threads_started":1}
I20260808 15:49:20.837061 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485a9918c78c4de5834d3ff1c6ed99aa" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" is_pre_election: true
I20260808 15:49:20.836966 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485a9918c78c4de5834d3ff1c6ed99aa" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90189dc1cb624f0ab23ac22ad88061ac" is_pre_election: true
I20260808 15:49:20.837389 8089 raft_consensus.cc:2454] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 0.
I20260808 15:49:20.837499 7871 raft_consensus.cc:2454] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 0.
I20260808 15:49:20.837952 7543 leader_election.cc:302] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b4e52b4d0444aaa9c6b3b01813da6d, 46e8a2ef7db640af8712eb1fcd33445c; no voters:
I20260808 15:49:20.838456 8218 raft_consensus.cc:2781] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:20.838596 8218 raft_consensus.cc:482] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:20.838726 8218 raft_consensus.cc:3037] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:20.840970 8218 raft_consensus.cc:504] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.841735 8218 leader_election.cc:288] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 election: Requested vote from peers 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:20.842218 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485a9918c78c4de5834d3ff1c6ed99aa" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90189dc1cb624f0ab23ac22ad88061ac"
I20260808 15:49:20.842293 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "485a9918c78c4de5834d3ff1c6ed99aa" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
I20260808 15:49:20.842454 7871 raft_consensus.cc:3037] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:20.842531 8089 raft_consensus.cc:3037] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:20.844848 8089 raft_consensus.cc:2454] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 1.
I20260808 15:49:20.845284 7543 leader_election.cc:302] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b4e52b4d0444aaa9c6b3b01813da6d, 46e8a2ef7db640af8712eb1fcd33445c; no voters:
I20260808 15:49:20.845594 8218 raft_consensus.cc:2781] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:20.845844 7871 raft_consensus.cc:2454] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 1.
I20260808 15:49:20.846246 8218 raft_consensus.cc:686] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 1 LEADER]: Becoming Leader. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Running, Role: LEADER
I20260808 15:49:20.846792 8218 consensus_queue.cc:237] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.851133 7412 catalog_manager.cc:5495] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193). New cstate: current_term: 1 leader_uuid: "46e8a2ef7db640af8712eb1fcd33445c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } health_report { overall_health: UNKNOWN } } }
Aug 08 15:49:20 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204157, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.254@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:20.925967 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.909470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.27.84.254:42387)
0808 15:49:20.909749 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:20.909757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:20.910381 (+ 624us) server_negotiation.cc:407] Connection header received
0808 15:49:20.910587 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:20.910591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:20.910676 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:20.910762 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:20.912415 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.913178 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.915112 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.915548 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.915651 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.919658 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:20.919665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:20.919670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:20.919707 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:20.923083 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:20.923809 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:20.923841 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:20.923851 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:20.923968 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:20.924370 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:20.924377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:20.924380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:20.924651 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:20.924801 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:20.925242 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 15:49:20.925585 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":131}
I20260808 15:49:20.925992 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:20.909525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42387 (local address 127.0.0.1:33288)
0808 15:49:20.910239 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:20.910267 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:20.910443 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:20.910831 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:20.910837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:20.911824 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:20.912202 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.912215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.913330 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.913338 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.914939 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:20.914956 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:20.915751 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:20.915758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:20.915918 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:20.916648 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:20.916678 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:20.919461 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:20.923203 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:20.923214 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:20.923221 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:20.923648 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:20.924095 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:20.924100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:20.924105 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:20.924255 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:20.924792 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:20.924801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:20.924964 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 15:49:20.925129 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 15:49:20.925444 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":387,"threads_started":1}
I20260808 15:49:20.931913 7412 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:33288:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260808 15:49:20.933038 7412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:20.940541 7837 tablet_service.cc:1467] Processing CreateTablet for tablet ae5e693bf3224de99e35579c04239f01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6720a81ad9340eb8bbb21399459c436]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:49:20.940500 7622 tablet_service.cc:1467] Processing CreateTablet for tablet ae5e693bf3224de99e35579c04239f01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6720a81ad9340eb8bbb21399459c436]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:49:20.940846 8049 tablet_service.cc:1467] Processing CreateTablet for tablet ae5e693bf3224de99e35579c04239f01 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e6720a81ad9340eb8bbb21399459c436]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260808 15:49:20.941097 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5e693bf3224de99e35579c04239f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.941118 7837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5e693bf3224de99e35579c04239f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.941394 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae5e693bf3224de99e35579c04239f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:20.949056 8213 tablet_bootstrap.cc:492] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: Bootstrap starting.
I20260808 15:49:20.949396 8210 tablet_bootstrap.cc:492] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: Bootstrap starting.
I20260808 15:49:20.950240 8211 tablet_bootstrap.cc:492] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: Bootstrap starting.
I20260808 15:49:20.950949 8210 tablet_bootstrap.cc:654] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.951916 8211 tablet_bootstrap.cc:654] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.953644 8210 tablet_bootstrap.cc:492] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: No bootstrap required, opened a new log
I20260808 15:49:20.953843 8210 ts_tablet_manager.cc:1397] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260808 15:49:20.954214 8213 tablet_bootstrap.cc:654] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:20.954551 8210 raft_consensus.cc:348] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.954878 8210 raft_consensus.cc:374] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.954991 8210 raft_consensus.cc:729] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.955305 8210 consensus_queue.cc:260] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.955991 8210 ts_tablet_manager.cc:1428] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:49:20.956476 8211 tablet_bootstrap.cc:492] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: No bootstrap required, opened a new log
I20260808 15:49:20.956638 8211 ts_tablet_manager.cc:1397] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260808 15:49:20.957106 8235 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:20.957479 8235 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:20.957465 8211 raft_consensus.cc:348] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.957724 8211 raft_consensus.cc:374] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.957804 8211 raft_consensus.cc:729] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b4e52b4d0444aaa9c6b3b01813da6d, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.958267 8211 consensus_queue.cc:260] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.958889 8221 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:20.959291 8221 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:20.959844 8211 ts_tablet_manager.cc:1428] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:20.966070 8213 tablet_bootstrap.cc:492] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: No bootstrap required, opened a new log
I20260808 15:49:20.966269 8213 ts_tablet_manager.cc:1397] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260808 15:49:20.967149 8213 raft_consensus.cc:348] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.967442 8213 raft_consensus.cc:374] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:20.967528 8213 raft_consensus.cc:729] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Initialized, Role: FOLLOWER
I20260808 15:49:20.967775 8213 consensus_queue.cc:260] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:20.968262 8218 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:20.968534 8218 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:20.969049 8213 ts_tablet_manager.cc:1428] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260808 15:49:20.987600 8172 tablet.cc:2369] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:21.235427 8235 raft_consensus.cc:482] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:21.235677 8235 raft_consensus.cc:504] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.236519 8235 leader_election.cc:288] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:21.237210 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5e693bf3224de99e35579c04239f01" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" is_pre_election: true
I20260808 15:49:21.237290 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5e693bf3224de99e35579c04239f01" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e8a2ef7db640af8712eb1fcd33445c" is_pre_election: true
I20260808 15:49:21.237501 8089 raft_consensus.cc:2454] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 0.
I20260808 15:49:21.237566 7648 raft_consensus.cc:2454] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 0.
I20260808 15:49:21.238023 7758 leader_election.cc:302] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b4e52b4d0444aaa9c6b3b01813da6d, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:21.238493 8235 raft_consensus.cc:2781] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:21.238660 8235 raft_consensus.cc:482] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:21.238785 8235 raft_consensus.cc:3037] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:21.240849 8235 raft_consensus.cc:504] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.241451 8235 leader_election.cc:288] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 election: Requested vote from peers 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:21.241987 7648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5e693bf3224de99e35579c04239f01" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e8a2ef7db640af8712eb1fcd33445c"
I20260808 15:49:21.242081 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5e693bf3224de99e35579c04239f01" candidate_uuid: "90189dc1cb624f0ab23ac22ad88061ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
I20260808 15:49:21.242209 7648 raft_consensus.cc:3037] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:21.242313 8089 raft_consensus.cc:3037] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:21.244395 7648 raft_consensus.cc:2454] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 1.
I20260808 15:49:21.244792 7758 leader_election.cc:302] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e8a2ef7db640af8712eb1fcd33445c, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:21.245085 8235 raft_consensus.cc:2781] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:21.245301 8089 raft_consensus.cc:2454] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90189dc1cb624f0ab23ac22ad88061ac in term 1.
I20260808 15:49:21.245642 8235 raft_consensus.cc:686] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [term 1 LEADER]: Becoming Leader. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Running, Role: LEADER
I20260808 15:49:21.245959 8235 consensus_queue.cc:237] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.247699 8268 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: TxnStatusTablet state changed. Reason: New leader 90189dc1cb624f0ab23ac22ad88061ac. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.248057 8268 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:49:21.248633 8273 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260808 15:49:21.248888 8273 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260808 15:49:21.249401 7412 catalog_manager.cc:5495] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194). New cstate: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260808 15:49:21.270545 8262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.264805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42303 (local address 127.0.0.1:44468)
0808 15:49:21.265072 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:21.265097 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:21.265236 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:21.265647 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:21.265650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:21.265661 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:49:21.265846 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:21.265853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.267377 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.267380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:21.268373 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:21.268385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.269353 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.269357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:21.269519 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.269575 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.270042 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.270067 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:49:21.270157 (+ 90us) client_negotiation.cc:240] Negotiation successful
0808 15:49:21.270297 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 15:49:21.270632 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.265007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.27.84.194:42303)
0808 15:49:21.265227 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.265234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.265255 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:49:21.265422 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.265428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.265513 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.265617 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:21.265996 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.267241 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.268517 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.269191 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.269343 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.269757 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.269890 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.269992 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:21.270212 (+ 220us) server_negotiation.cc:299] Negotiation successful
0808 15:49:21.270297 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:49:21.275321 8089 raft_consensus.cc:1261] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Refusing update from remote peer 90189dc1cb624f0ab23ac22ad88061ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:21.275429 7648 raft_consensus.cc:1261] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Refusing update from remote peer 90189dc1cb624f0ab23ac22ad88061ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:21.276094 8268 consensus_queue.cc:1035] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:21.276430 8235 consensus_queue.cc:1035] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:21.281735 8221 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: TxnStatusTablet state changed. Reason: New leader 90189dc1cb624f0ab23ac22ad88061ac. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.282039 8221 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:21.282398 8250 consensus_queue.cc:1035] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260808 15:49:21.287022 8218 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: TxnStatusTablet state changed. Reason: New leader 90189dc1cb624f0ab23ac22ad88061ac. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.287362 8218 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:21.289911 8268 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.290236 8268 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:49:21.290427 8283 mvcc.cc:205] Tried to move back new op lower bound from 7316292244576276480 to 7316292244465700864. Current Snapshot: MvccSnapshot[applied={T|T < 7316292244576276480}]
I20260808 15:49:21.290673 8268 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.290910 8268 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 90189dc1cb624f0ab23ac22ad88061ac: This TxnStatusTablet replica's current role is: LEADER
I20260808 15:49:21.292523 8221 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.292798 8221 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 37b4e52b4d0444aaa9c6b3b01813da6d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:21.295282 8218 tablet_replica.cc:439] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90189dc1cb624f0ab23ac22ad88061ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } } }
I20260808 15:49:21.295626 8218 tablet_replica.cc:442] T ae5e693bf3224de99e35579c04239f01 P 46e8a2ef7db640af8712eb1fcd33445c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260808 15:49:21.303098 8218 consensus_queue.cc:1035] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:21.317380 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.308728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49594 (local address 127.27.84.193:38349)
0808 15:49:21.310652 (+ 1924us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.310659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.311072 (+ 413us) server_negotiation.cc:407] Connection header received
0808 15:49:21.311238 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.311243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.311365 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.311552 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:21.312019 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.313209 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.314650 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.315332 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.315465 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.315913 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.316680 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.316835 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:21.316919 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 15:49:21.317005 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1733,"thread_start_us":1594,"threads_started":1}
I20260808 15:49:21.334841 8205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.329026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:34005 (local address 127.27.84.194:42303)
0808 15:49:21.329265 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.329273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.329745 (+ 472us) server_negotiation.cc:407] Connection header received
0808 15:49:21.329987 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.329991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.330058 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.330161 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:21.330677 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.331488 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.332923 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.333380 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.333551 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.333965 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.334110 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.334215 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:21.334365 (+ 150us) server_negotiation.cc:299] Negotiation successful
0808 15:49:21.334454 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:49:21.334916 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.329022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42303 (local address 127.27.84.193:34005)
0808 15:49:21.329636 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:21.329652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:21.329831 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:21.330175 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:21.330180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:21.330192 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:49:21.330499 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:21.330507 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.331641 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.331647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:21.332768 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:21.332786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.333566 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.333572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:21.333715 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.333769 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.334229 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.334243 (+ 14us) client_negotiation.cc:769] Sending connection context
0808 15:49:21.334341 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:49:21.334479 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":347,"threads_started":1}
I20260808 15:49:21.380329 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.372802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:44593 (local address 127.27.84.193:38349)
0808 15:49:21.372982 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.372990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.374013 (+ 1023us) server_negotiation.cc:407] Connection header received
0808 15:49:21.374210 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.374216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.374309 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.374407 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:21.375157 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.376085 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.378155 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.378776 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.378910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.379534 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.379679 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.379763 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:21.380027 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:49:21.380094 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:49:21.380640 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.372858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:38349 (local address 127.27.84.194:44593)
0808 15:49:21.373886 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:21.373915 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:21.374062 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:21.374444 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:21.374448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:21.374463 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0808 15:49:21.374987 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:21.375001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.376226 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.376232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:21.378015 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:21.378028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.378997 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.379006 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:21.379206 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.379297 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.379862 (+ 565us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.379891 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:49:21.379983 (+ 92us) client_negotiation.cc:240] Negotiation successful
0808 15:49:21.380159 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":624,"threads_started":1}
I20260808 15:49:21.434024 7412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:21.434293 7412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:21.438021 7412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Sending DeleteTablet for 3 replicas of tablet 485a9918c78c4de5834d3ff1c6ed99aa
I20260808 15:49:21.438901 7622 tablet_service.cc:1514] Processing DeleteTablet for tablet 485a9918c78c4de5834d3ff1c6ed99aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:21 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49586
I20260808 15:49:21.439049 7837 tablet_service.cc:1514] Processing DeleteTablet for tablet 485a9918c78c4de5834d3ff1c6ed99aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:21 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:44448
I20260808 15:49:21.439958 8315 tablet_replica.cc:330] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: stopping tablet replica
I20260808 15:49:21.440125 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 485a9918c78c4de5834d3ff1c6ed99aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:21 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45060
I20260808 15:49:21.440410 8315 raft_consensus.cc:2229] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:21.441560 8315 raft_consensus.cc:2258] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:21.443262 8315 ts_tablet_manager.cc:1905] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:21.452571 8315 ts_tablet_manager.cc:1918] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:49:21.452850 8315 log.cc:1196] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/485a9918c78c4de5834d3ff1c6ed99aa
I20260808 15:49:21.453709 8315 ts_tablet_manager.cc:1939] T 485a9918c78c4de5834d3ff1c6ed99aa P 90189dc1cb624f0ab23ac22ad88061ac: Deleting consensus metadata
I20260808 15:49:21.455109 7391 catalog_manager.cc:4841] TS 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303): tablet 485a9918c78c4de5834d3ff1c6ed99aa (table test-table [id=5a66fcf9d63844f89dd04e01df4f621c]) successfully deleted
I20260808 15:49:21.459203 8313 tablet_replica.cc:330] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: stopping tablet replica
I20260808 15:49:21.459530 8313 raft_consensus.cc:2229] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:21.460731 8313 raft_consensus.cc:2258] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:21.462231 8313 ts_tablet_manager.cc:1905] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:21.465070 8317 tablet_replica.cc:330] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: stopping tablet replica
I20260808 15:49:21.465412 8317 raft_consensus.cc:2229] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:21.465776 8317 raft_consensus.cc:2258] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:21.467062 8317 ts_tablet_manager.cc:1905] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:21.471071 8313 ts_tablet_manager.cc:1918] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:49:21.471239 8313 log.cc:1196] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/485a9918c78c4de5834d3ff1c6ed99aa
I20260808 15:49:21.471798 8313 ts_tablet_manager.cc:1939] T 485a9918c78c4de5834d3ff1c6ed99aa P 46e8a2ef7db640af8712eb1fcd33445c: Deleting consensus metadata
I20260808 15:49:21.473179 7388 catalog_manager.cc:4841] TS 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349): tablet 485a9918c78c4de5834d3ff1c6ed99aa (table test-table [id=5a66fcf9d63844f89dd04e01df4f621c]) successfully deleted
I20260808 15:49:21.474700 8317 ts_tablet_manager.cc:1918] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:49:21.474923 8317 log.cc:1196] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/485a9918c78c4de5834d3ff1c6ed99aa
I20260808 15:49:21.475438 8317 ts_tablet_manager.cc:1939] T 485a9918c78c4de5834d3ff1c6ed99aa P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting consensus metadata
I20260808 15:49:21.476764 7388 catalog_manager.cc:4841] TS 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133): tablet 485a9918c78c4de5834d3ff1c6ed99aa (table test-table [id=5a66fcf9d63844f89dd04e01df4f621c]) successfully deleted
Aug 08 15:49:21 dist-test-slave-twwt krb5kdc[7348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 08 15:49:21 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:21 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:49:21.551005 8327 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:21.535956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.27.84.254:42387)
0808 15:49:21.536600 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.536607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.537313 (+ 706us) server_negotiation.cc:407] Connection header received
0808 15:49:21.537517 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.537524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.537630 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.537775 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:21.539613 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.540915 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.542864 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.543551 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.543692 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.550663 (+ 6971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33294: BlockingRecv error: recv got EOF from 127.0.0.1:33294 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":263,"threads_started":1}
Aug 08 15:49:21 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM
I20260808 15:49:21.576547 8327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.556551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.27.84.254:42387)
0808 15:49:21.556774 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.556781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.557341 (+ 560us) server_negotiation.cc:407] Connection header received
0808 15:49:21.557618 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.557624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.557709 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.557782 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:21.559609 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.560759 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.562875 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.563563 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.563701 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.569247 (+ 5546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:21.569256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:21.569261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:21.569315 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:21.573212 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:21.574176 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:21.574183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:21.574186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:21.574372 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:21.574910 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:21.574916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:21.574919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:21.575302 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:21.575575 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:21.575914 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:49:21.576161 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:49:21.581053 7412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:21.581925 7412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:21.590426 7837 tablet_service.cc:1467] Processing CreateTablet for tablet 4a033420414648f294fe4952a7da696e (DEFAULT_TABLE table=test-table [id=f153c70ae4584912896416f775d45e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:21.590898 7622 tablet_service.cc:1467] Processing CreateTablet for tablet 4a033420414648f294fe4952a7da696e (DEFAULT_TABLE table=test-table [id=f153c70ae4584912896416f775d45e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:21.591053 7837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a033420414648f294fe4952a7da696e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:21.591388 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a033420414648f294fe4952a7da696e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:21.598965 8049 tablet_service.cc:1467] Processing CreateTablet for tablet 4a033420414648f294fe4952a7da696e (DEFAULT_TABLE table=test-table [id=f153c70ae4584912896416f775d45e29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:21.599476 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a033420414648f294fe4952a7da696e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:21.615917 8337 tablet_bootstrap.cc:492] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Bootstrap starting.
I20260808 15:49:21.618202 8335 tablet_bootstrap.cc:492] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Bootstrap starting.
I20260808 15:49:21.619688 8337 tablet_bootstrap.cc:654] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:21.621475 8335 tablet_bootstrap.cc:654] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:21.625633 8337 tablet_bootstrap.cc:492] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: No bootstrap required, opened a new log
I20260808 15:49:21.625910 8337 ts_tablet_manager.cc:1397] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260808 15:49:21.626868 8337 raft_consensus.cc:348] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.627799 8336 tablet_bootstrap.cc:492] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Bootstrap starting.
I20260808 15:49:21.627959 8337 raft_consensus.cc:374] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:21.628109 8337 raft_consensus.cc:729] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b4e52b4d0444aaa9c6b3b01813da6d, State: Initialized, Role: FOLLOWER
I20260808 15:49:21.628448 8337 consensus_queue.cc:260] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.630884 8336 tablet_bootstrap.cc:654] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:21.631116 8337 ts_tablet_manager.cc:1428] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260808 15:49:21.633026 8335 tablet_bootstrap.cc:492] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: No bootstrap required, opened a new log
I20260808 15:49:21.633206 8335 ts_tablet_manager.cc:1397] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20260808 15:49:21.634171 8335 raft_consensus.cc:348] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.634469 8335 raft_consensus.cc:374] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:21.634578 8335 raft_consensus.cc:729] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Initialized, Role: FOLLOWER
I20260808 15:49:21.634904 8335 consensus_queue.cc:260] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.636071 8335 ts_tablet_manager.cc:1428] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:21.641784 8336 tablet_bootstrap.cc:492] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: No bootstrap required, opened a new log
I20260808 15:49:21.642014 8336 ts_tablet_manager.cc:1397] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20260808 15:49:21.643038 8336 raft_consensus.cc:348] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.643316 8336 raft_consensus.cc:374] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:21.643427 8336 raft_consensus.cc:729] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Initialized, Role: FOLLOWER
I20260808 15:49:21.643716 8336 consensus_queue.cc:260] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.644407 8336 ts_tablet_manager.cc:1428] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260808 15:49:21.670701 8309 raft_consensus.cc:482] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:21.670996 8309 raft_consensus.cc:504] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.671939 8309 leader_election.cc:288] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:21.672976 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a033420414648f294fe4952a7da696e" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90189dc1cb624f0ab23ac22ad88061ac" is_pre_election: true
I20260808 15:49:21.673290 7871 raft_consensus.cc:2454] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 0.
I20260808 15:49:21.673943 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a033420414648f294fe4952a7da696e" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" is_pre_election: true
I20260808 15:49:21.674237 8089 raft_consensus.cc:2454] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 0.
I20260808 15:49:21.674800 7543 leader_election.cc:302] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37b4e52b4d0444aaa9c6b3b01813da6d, 46e8a2ef7db640af8712eb1fcd33445c; no voters:
I20260808 15:49:21.675555 8309 raft_consensus.cc:2781] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:21.675709 8309 raft_consensus.cc:482] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:21.675817 8309 raft_consensus.cc:3037] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:21.680912 8309 raft_consensus.cc:504] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.682120 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a033420414648f294fe4952a7da696e" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90189dc1cb624f0ab23ac22ad88061ac"
I20260808 15:49:21.682433 7871 raft_consensus.cc:3037] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:21.686236 7871 raft_consensus.cc:2454] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 1.
I20260808 15:49:21.686812 7546 leader_election.cc:302] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e8a2ef7db640af8712eb1fcd33445c, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:21.687217 8304 raft_consensus.cc:2781] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:21.687397 8304 raft_consensus.cc:686] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 1 LEADER]: Becoming Leader. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Running, Role: LEADER
I20260808 15:49:21.687748 8304 consensus_queue.cc:237] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.691672 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a033420414648f294fe4952a7da696e" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
I20260808 15:49:21.691942 8089 raft_consensus.cc:3037] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:21.695020 8089 raft_consensus.cc:2454] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 1.
I20260808 15:49:21.695881 8309 leader_election.cc:288] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 election: Requested vote from peers 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:21.703378 7408 catalog_manager.cc:5495] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193). New cstate: current_term: 1 leader_uuid: "46e8a2ef7db640af8712eb1fcd33445c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260808 15:49:21.738813 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:21.731363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.27.84.193:38349)
0808 15:49:21.731578 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:21.731584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:21.731859 (+ 275us) server_negotiation.cc:407] Connection header received
0808 15:49:21.732103 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:21.732109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:21.732239 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:21.732350 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:21.733009 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.734323 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.736448 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:21.737203 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:21.737352 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:21.737887 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:21.738036 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:21.738138 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:21.738324 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 15:49:21.738419 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:49:21.742509 8089 raft_consensus.cc:1261] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Refusing update from remote peer 46e8a2ef7db640af8712eb1fcd33445c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:21.742523 7871 raft_consensus.cc:1261] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Refusing update from remote peer 46e8a2ef7db640af8712eb1fcd33445c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:21.743423 8309 consensus_queue.cc:1035] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:21.743757 8218 consensus_queue.cc:1035] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:21.771710 7408 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:21.771996 7408 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:21.775710 7408 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Sending DeleteTablet for 3 replicas of tablet 4a033420414648f294fe4952a7da696e
I20260808 15:49:21.776729 7622 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a033420414648f294fe4952a7da696e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:21 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49586
I20260808 15:49:21.776804 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a033420414648f294fe4952a7da696e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:21 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45060
I20260808 15:49:21.777122 8317 tablet_replica.cc:330] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: stopping tablet replica
I20260808 15:49:21.777136 8313 tablet_replica.cc:330] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: stopping tablet replica
I20260808 15:49:21.777364 8317 raft_consensus.cc:2229] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:21.777408 8313 raft_consensus.cc:2229] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:21.777623 8317 raft_consensus.cc:2258] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:21.777822 8313 raft_consensus.cc:2258] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:21.778704 7837 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a033420414648f294fe4952a7da696e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:21 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:44448
I20260808 15:49:21.779402 8317 ts_tablet_manager.cc:1905] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:21.779927 7408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:21.780766 7408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:21.782580 8315 tablet_replica.cc:330] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: stopping tablet replica
I20260808 15:49:21.782841 8315 raft_consensus.cc:2229] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:21.783121 8315 raft_consensus.cc:2258] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:21.785207 8313 ts_tablet_manager.cc:1905] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:21.785287 8315 ts_tablet_manager.cc:1905] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:21.792575 8315 ts_tablet_manager.cc:1918] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:21.792860 8315 log.cc:1196] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/4a033420414648f294fe4952a7da696e
I20260808 15:49:21.793574 8315 ts_tablet_manager.cc:1939] T 4a033420414648f294fe4952a7da696e P 90189dc1cb624f0ab23ac22ad88061ac: Deleting consensus metadata
I20260808 15:49:21.795076 7391 catalog_manager.cc:4841] TS 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303): tablet 4a033420414648f294fe4952a7da696e (table test-table [id=f153c70ae4584912896416f775d45e29]) successfully deleted
I20260808 15:49:21.797415 8313 ts_tablet_manager.cc:1918] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:21.797596 8313 log.cc:1196] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/4a033420414648f294fe4952a7da696e
I20260808 15:49:21.798100 8313 ts_tablet_manager.cc:1939] T 4a033420414648f294fe4952a7da696e P 46e8a2ef7db640af8712eb1fcd33445c: Deleting consensus metadata
I20260808 15:49:21.799563 7388 catalog_manager.cc:4841] TS 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349): tablet 4a033420414648f294fe4952a7da696e (table test-table [id=f153c70ae4584912896416f775d45e29]) successfully deleted
I20260808 15:49:21.801971 8317 ts_tablet_manager.cc:1918] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:21.802151 8317 log.cc:1196] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/4a033420414648f294fe4952a7da696e
I20260808 15:49:21.802919 8317 ts_tablet_manager.cc:1939] T 4a033420414648f294fe4952a7da696e P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting consensus metadata
I20260808 15:49:21.804168 7388 catalog_manager.cc:4841] TS 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133): tablet 4a033420414648f294fe4952a7da696e (table test-table [id=f153c70ae4584912896416f775d45e29]) successfully deleted
I20260808 15:49:21.805399 7622 tablet_service.cc:1467] Processing CreateTablet for tablet a35289a6c269418f8cd8cba990a0171c (DEFAULT_TABLE table=test-table [id=2c4f6f94d7d349a3861752933b5ea128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:21.805688 7837 tablet_service.cc:1467] Processing CreateTablet for tablet a35289a6c269418f8cd8cba990a0171c (DEFAULT_TABLE table=test-table [id=2c4f6f94d7d349a3861752933b5ea128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:21.805897 7622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35289a6c269418f8cd8cba990a0171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:21.806115 7837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35289a6c269418f8cd8cba990a0171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:21.810575 8049 tablet_service.cc:1467] Processing CreateTablet for tablet a35289a6c269418f8cd8cba990a0171c (DEFAULT_TABLE table=test-table [id=2c4f6f94d7d349a3861752933b5ea128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:21.811084 8049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35289a6c269418f8cd8cba990a0171c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:21.814064 8336 tablet_bootstrap.cc:492] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Bootstrap starting.
I20260808 15:49:21.814137 8335 tablet_bootstrap.cc:492] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Bootstrap starting.
I20260808 15:49:21.815866 8336 tablet_bootstrap.cc:654] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:21.816009 8335 tablet_bootstrap.cc:654] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:21.819644 8337 tablet_bootstrap.cc:492] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Bootstrap starting.
I20260808 15:49:21.822105 8337 tablet_bootstrap.cc:654] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:21.827870 8335 tablet_bootstrap.cc:492] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: No bootstrap required, opened a new log
I20260808 15:49:21.828150 8335 ts_tablet_manager.cc:1397] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260808 15:49:21.829470 8335 raft_consensus.cc:348] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.829877 8335 raft_consensus.cc:374] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:21.829981 8335 raft_consensus.cc:729] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Initialized, Role: FOLLOWER
I20260808 15:49:21.830394 8335 consensus_queue.cc:260] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.831208 8335 ts_tablet_manager.cc:1428] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260808 15:49:21.833482 8336 tablet_bootstrap.cc:492] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: No bootstrap required, opened a new log
I20260808 15:49:21.833704 8336 ts_tablet_manager.cc:1397] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260808 15:49:21.833945 8337 tablet_bootstrap.cc:492] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: No bootstrap required, opened a new log
I20260808 15:49:21.834143 8337 ts_tablet_manager.cc:1397] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260808 15:49:21.834681 8336 raft_consensus.cc:348] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.835000 8336 raft_consensus.cc:374] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:21.835126 8336 raft_consensus.cc:729] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90189dc1cb624f0ab23ac22ad88061ac, State: Initialized, Role: FOLLOWER
I20260808 15:49:21.835037 8337 raft_consensus.cc:348] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.835332 8337 raft_consensus.cc:374] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:21.835422 8337 raft_consensus.cc:729] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b4e52b4d0444aaa9c6b3b01813da6d, State: Initialized, Role: FOLLOWER
I20260808 15:49:21.835510 8336 consensus_queue.cc:260] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.835677 8337 consensus_queue.cc:260] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:21.836411 8337 ts_tablet_manager.cc:1428] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260808 15:49:21.836925 8336 ts_tablet_manager.cc:1428] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:22.021457 8218 raft_consensus.cc:482] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:22.021749 8218 raft_consensus.cc:504] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:22.022526 8218 leader_election.cc:288] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:22.023191 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35289a6c269418f8cd8cba990a0171c" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90189dc1cb624f0ab23ac22ad88061ac" is_pre_election: true
I20260808 15:49:22.023372 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35289a6c269418f8cd8cba990a0171c" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" is_pre_election: true
I20260808 15:49:22.023524 7871 raft_consensus.cc:2454] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 0.
I20260808 15:49:22.023664 8089 raft_consensus.cc:2454] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 0.
I20260808 15:49:22.024159 7546 leader_election.cc:302] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 37b4e52b4d0444aaa9c6b3b01813da6d, 46e8a2ef7db640af8712eb1fcd33445c, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:22.024530 8218 raft_consensus.cc:2781] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:22.024649 8218 raft_consensus.cc:482] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:22.024792 8218 raft_consensus.cc:3037] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:22.027001 8218 raft_consensus.cc:504] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:22.027743 8218 leader_election.cc:288] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 election: Requested vote from peers 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303), 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133)
I20260808 15:49:22.028237 7871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35289a6c269418f8cd8cba990a0171c" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90189dc1cb624f0ab23ac22ad88061ac"
I20260808 15:49:22.028375 8089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35289a6c269418f8cd8cba990a0171c" candidate_uuid: "46e8a2ef7db640af8712eb1fcd33445c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d"
I20260808 15:49:22.028508 7871 raft_consensus.cc:3037] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:22.028626 8089 raft_consensus.cc:3037] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:22.031683 8089 raft_consensus.cc:2454] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 1.
I20260808 15:49:22.031683 7871 raft_consensus.cc:2454] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e8a2ef7db640af8712eb1fcd33445c in term 1.
I20260808 15:49:22.032272 7546 leader_election.cc:302] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e8a2ef7db640af8712eb1fcd33445c, 90189dc1cb624f0ab23ac22ad88061ac; no voters:
I20260808 15:49:22.032583 8218 raft_consensus.cc:2781] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:22.032789 8218 raft_consensus.cc:686] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 1 LEADER]: Becoming Leader. State: Replica: 46e8a2ef7db640af8712eb1fcd33445c, State: Running, Role: LEADER
I20260808 15:49:22.033077 8218 consensus_queue.cc:237] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } }
I20260808 15:49:22.035877 7412 catalog_manager.cc:5495] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193). New cstate: current_term: 1 leader_uuid: "46e8a2ef7db640af8712eb1fcd33445c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46e8a2ef7db640af8712eb1fcd33445c" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 38349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 } health_report { overall_health: UNKNOWN } } }
I20260808 15:49:22.083985 8089 raft_consensus.cc:1261] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Refusing update from remote peer 46e8a2ef7db640af8712eb1fcd33445c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:22.083956 7871 raft_consensus.cc:1261] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Refusing update from remote peer 46e8a2ef7db640af8712eb1fcd33445c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:22.084745 8218 consensus_queue.cc:1035] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90189dc1cb624f0ab23ac22ad88061ac" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:22.085237 8309 consensus_queue.cc:1035] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b4e52b4d0444aaa9c6b3b01813da6d" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:22.140949 7410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:22.141213 7410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:22.144114 7410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09142f0c985d49539e62db93b42a53c0: Sending DeleteTablet for 3 replicas of tablet a35289a6c269418f8cd8cba990a0171c
I20260808 15:49:22.144958 7622 tablet_service.cc:1514] Processing DeleteTablet for tablet a35289a6c269418f8cd8cba990a0171c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:22 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49586
I20260808 15:49:22.145108 8049 tablet_service.cc:1514] Processing DeleteTablet for tablet a35289a6c269418f8cd8cba990a0171c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:22 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45060
I20260808 15:49:22.145264 8313 tablet_replica.cc:330] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: stopping tablet replica
I20260808 15:49:22.145228 7837 tablet_service.cc:1514] Processing DeleteTablet for tablet a35289a6c269418f8cd8cba990a0171c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:22 UTC) from {username='oryx', principal='oryx/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:44448
I20260808 15:49:22.145426 8317 tablet_replica.cc:330] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: stopping tablet replica
I20260808 15:49:22.145442 8313 raft_consensus.cc:2229] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:22.145574 8317 raft_consensus.cc:2229] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:22.145731 8317 raft_consensus.cc:2258] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:22.145781 8313 raft_consensus.cc:2258] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:22.146106 8315 tablet_replica.cc:330] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: stopping tablet replica
I20260808 15:49:22.146306 8315 raft_consensus.cc:2229] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:22.146533 8315 raft_consensus.cc:2258] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:22.147226 8317 ts_tablet_manager.cc:1905] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:22.147228 8313 ts_tablet_manager.cc:1905] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:22.147850 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 7517
I20260808 15:49:22.147976 8315 ts_tablet_manager.cc:1905] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:22.154439 8317 ts_tablet_manager.cc:1918] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:49:22.154439 8313 ts_tablet_manager.cc:1918] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:49:22.154582 8313 log.cc:1196] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/a35289a6c269418f8cd8cba990a0171c
I20260808 15:49:22.154892 8317 log.cc:1196] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/a35289a6c269418f8cd8cba990a0171c
I20260808 15:49:22.155067 8313 ts_tablet_manager.cc:1939] T a35289a6c269418f8cd8cba990a0171c P 46e8a2ef7db640af8712eb1fcd33445c: Deleting consensus metadata
I20260808 15:49:22.155462 8317 ts_tablet_manager.cc:1939] T a35289a6c269418f8cd8cba990a0171c P 37b4e52b4d0444aaa9c6b3b01813da6d: Deleting consensus metadata
I20260808 15:49:22.156387 7388 catalog_manager.cc:4841] TS 46e8a2ef7db640af8712eb1fcd33445c (127.27.84.193:38349): tablet a35289a6c269418f8cd8cba990a0171c (table test-table [id=2c4f6f94d7d349a3861752933b5ea128]) successfully deleted
I20260808 15:49:22.156724 7388 catalog_manager.cc:4841] TS 37b4e52b4d0444aaa9c6b3b01813da6d (127.27.84.195:39133): tablet a35289a6c269418f8cd8cba990a0171c (table test-table [id=2c4f6f94d7d349a3861752933b5ea128]) successfully deleted
I20260808 15:49:22.157585 8315 ts_tablet_manager.cc:1918] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260808 15:49:22.157724 8315 log.cc:1196] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/a35289a6c269418f8cd8cba990a0171c
I20260808 15:49:22.158131 8315 ts_tablet_manager.cc:1939] T a35289a6c269418f8cd8cba990a0171c P 90189dc1cb624f0ab23ac22ad88061ac: Deleting consensus metadata
I20260808 15:49:22.159125 7391 catalog_manager.cc:4841] TS 90189dc1cb624f0ab23ac22ad88061ac (127.27.84.194:42303): tablet a35289a6c269418f8cd8cba990a0171c (table test-table [id=2c4f6f94d7d349a3861752933b5ea128]) successfully deleted
Aug 08 15:49:22 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.193@KRBTEST.COM
I20260808 15:49:22.171522 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:22.149345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49608 (local address 127.27.84.193:38349)
0808 15:49:22.149530 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:22.149536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:22.157074 (+ 7538us) server_negotiation.cc:407] Connection header received
0808 15:49:22.157204 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:22.157209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:22.157319 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:22.157456 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:22.159053 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:22.159870 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:22.161048 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:22.161430 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:22.161522 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:22.165852 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:22.165890 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:22.165907 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:22.165945 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:22.168927 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:22.169565 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:22.169573 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:22.169575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:22.169717 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:22.170057 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:22.170060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:22.170062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:22.170403 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:22.170715 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:22.171112 (+ 397us) server_negotiation.cc:299] Negotiation successful
0808 15:49:22.171276 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:49:22.172174 7718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:22.303589 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 7517
I20260808 15:49:22.340008 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 7732
Aug 08 15:49:22 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.194@KRBTEST.COM
I20260808 15:49:22.355721 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:22.340796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.27.84.194:42303)
0808 15:49:22.341224 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:22.341228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:22.341254 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:49:22.341316 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:22.341320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:22.341387 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:22.341478 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:22.342967 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:22.343681 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:22.344886 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:22.345274 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:22.345378 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:22.349677 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:22.349691 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:22.349707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:22.349740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:22.352755 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:22.353519 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:22.353527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:22.353530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:22.353667 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:22.354013 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:22.354017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:22.354019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:22.354388 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:22.354702 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:22.355151 (+ 449us) server_negotiation.cc:299] Negotiation successful
0808 15:49:22.355332 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260808 15:49:22.356415 7938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:22.509512 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 7732
I20260808 15:49:22.550114 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 7954
Aug 08 15:49:22 dist-test-slave-twwt krb5kdc[7348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.84.195@KRBTEST.COM
I20260808 15:49:22.567502 8375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:22.551014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.27.84.195:39133)
0808 15:49:22.551522 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:22.551526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:22.551544 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:49:22.551603 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:22.551607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:22.551683 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:22.551840 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:22.553294 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:22.554087 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:22.555396 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:22.555852 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:22.555951 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:22.560580 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:22.560599 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:22.560617 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:22.560670 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:22.563992 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:22.564710 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:22.564718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:22.564721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:22.565121 (+ 400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:22.565508 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:22.565512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:22.565514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:22.566002 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:22.566242 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:22.566602 (+ 360us) server_negotiation.cc:299] Negotiation successful
0808 15:49:22.567016 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":295,"threads_started":1}
I20260808 15:49:22.568245 8160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:22.668668 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 7954
I20260808 15:49:22.697057 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 7366
I20260808 15:49:22.698356 7471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:22.821305 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 7366
2026-08-08T15:49:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7559 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:22 dist-test-slave-twwt krb5kdc[8382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:22 dist-test-slave-twwt krb5kdc[8382](info): set up 2 sockets
Aug 08 15:49:22 dist-test-slave-twwt krb5kdc[8382](info): commencing operation
krb5kdc: starting...
W20260808 15:49:24.211943 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:24 dist-test-slave-twwt krb5kdc[8382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:49:24.506523 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:35997
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:24.862412 8405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:24.862777 8405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:24.862875 8405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:24.871702 8405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:24.871820 8405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:24.871867 8405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:24.871932 8405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:24.871980 8405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:24.882259 8405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:24.883693 8405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:24.885022 8405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:24.898536 8414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:24.898577 8411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:24.898487 8410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:24.899892 8405 server_base.cc:1034] running on GCE node
I20260808 15:49:24.900736 8405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:24.902259 8405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:24.903458 8405 hybrid_clock.cc:648] HybridClock initialized: now 1786204164903366 us; error 81 us; skew 500 ppm
I20260808 15:49:24.903820 8405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-08T15:49:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2140 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:25 dist-test-slave-twwt krb5kdc[8421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:25 dist-test-slave-twwt krb5kdc[8421](info): set up 2 sockets
Aug 08 15:49:25 dist-test-slave-twwt krb5kdc[8421](info): commencing operation
krb5kdc: starting...
W20260808 15:49:26.306016 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:26 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:49:26.602865 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44445
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:44445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:26.957682 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:26.958052 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:26.958227 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:26.965507 8447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:26.965688 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:26.965811 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:26.965919 8447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:26.965991 8447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:26.976854 8447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:44445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:26.978334 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:26.979841 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:26.992429 8454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:26.992512 8460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:26.992544 8455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:26.993618 8447 server_base.cc:1034] running on GCE node
I20260808 15:49:26.994520 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:26.996016 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:26.997236 8447 hybrid_clock.cc:648] HybridClock initialized: now 1786204166997181 us; error 33 us; skew 500 ppm
I20260808 15:49:26.997651 8447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:26 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204166, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:27.000881 8447 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:49:27.002472 8447 webserver.cc:460] Webserver started at http://127.27.84.254:44767/ using document root <none> and password file <none>
I20260808 15:49:27.003229 8447 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:27.003341 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:27.003610 8447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:27.006706 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "7eb77613fd8f468f81f007c0460b2ada"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "9581e38b144a52bd6a82dc299df48a71"
server_key_iv: "7f22c4db7ac398675f8b4a086b2d6dd5"
server_key_version: "encryptionkey@0"
I20260808 15:49:27.007478 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "7eb77613fd8f468f81f007c0460b2ada"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "9581e38b144a52bd6a82dc299df48a71"
server_key_iv: "7f22c4db7ac398675f8b4a086b2d6dd5"
server_key_version: "encryptionkey@0"
I20260808 15:49:27.012980 8447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:27.016511 8469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:27.017926 8447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:49:27.018152 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "7eb77613fd8f468f81f007c0460b2ada"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "9581e38b144a52bd6a82dc299df48a71"
server_key_iv: "7f22c4db7ac398675f8b4a086b2d6dd5"
server_key_version: "encryptionkey@0"
I20260808 15:49:27.018380 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:27.039294 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:27.043715 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:27.043968 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:27.063503 8447 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:44445
I20260808 15:49:27.063535 8547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:44445 every 8 connection(s)
I20260808 15:49:27.064970 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:27.068950 8550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:27.073973 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 8447
I20260808 15:49:27.074193 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:27.075045 27987 external_mini_cluster.cc:1428] Setting key bfabc9a13e60789740a8f603b7dea05b
I20260808 15:49:27.081151 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Bootstrap starting.
I20260808 15:49:27.084333 8550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:27.085345 8550 log.cc:824] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:27.088125 8550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: No bootstrap required, opened a new log
Aug 08 15:49:27 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:27.096099 8550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:27.096518 8550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:27.096663 8550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb77613fd8f468f81f007c0460b2ada, State: Initialized, Role: FOLLOWER
I20260808 15:49:27.097352 8550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:27.097633 8550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:27.097793 8550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:27.097986 8550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:27.097991 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:27.077308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.27.84.254:44445)
0808 15:49:27.078243 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:27.078255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:27.078297 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:49:27.080122 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:27.080146 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:27.080518 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:27.080900 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:27.082456 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:27.083819 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:27.085142 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:27.085863 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:27.086072 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:27.090266 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:27.090274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:27.090296 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:27.090336 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:27.093620 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:27.094364 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:27.094380 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:27.094390 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:27.094574 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:27.095016 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:27.095025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:27.095029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:27.095563 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:27.095855 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:27.096338 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 15:49:27.096802 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":397,"threads_started":1}
I20260808 15:49:27.100905 8550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:27.101495 8550 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb77613fd8f468f81f007c0460b2ada; no voters:
I20260808 15:49:27.102070 8550 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:27.102221 8556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:27.103091 8556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 LEADER]: Becoming Leader. State: Replica: 7eb77613fd8f468f81f007c0460b2ada, State: Running, Role: LEADER
I20260808 15:49:27.103689 8556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:27.104318 8550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:27.106403 8560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb77613fd8f468f81f007c0460b2ada. Latest consensus state: current_term: 1 leader_uuid: "7eb77613fd8f468f81f007c0460b2ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } } }
I20260808 15:49:27.106851 8560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:27.107393 8557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb77613fd8f468f81f007c0460b2ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } } }
I20260808 15:49:27.107780 8557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:27.107793 8564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:27.113169 8564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:27.123248 8564 catalog_manager.cc:1347] Generated new cluster ID: 5d0a6bb118c644b68d96acb2a7d9ac09
I20260808 15:49:27.123375 8564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:27.131620 8564 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:27.132946 8564 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:27.150507 8564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Generated new TSK 0
I20260808 15:49:27.151424 8564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:49:27.234227 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44445
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:27.616971 8587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:27.617276 8587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:27.617376 8587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:27.624825 8587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:27.624949 8587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:27.625073 8587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:27.635149 8587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:27.636518 8587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:27.637873 8587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:27.652745 8593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:27.652804 8596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:27.652829 8594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:27.656219 8587 server_base.cc:1034] running on GCE node
I20260808 15:49:27.656886 8587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:27.657932 8587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:27.659196 8587 hybrid_clock.cc:648] HybridClock initialized: now 1786204167659133 us; error 75 us; skew 500 ppm
I20260808 15:49:27.659564 8587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:27 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204167, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:27.663111 8587 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:49:27.664734 8587 webserver.cc:460] Webserver started at http://127.27.84.193:33979/ using document root <none> and password file <none>
I20260808 15:49:27.665455 8587 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:27.665561 8587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:27.665838 8587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:27.669319 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "a5fcb63d23c34a63b7dae6e202fad886"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "13998a43a38806e006d61a5c47b5ba40"
server_key_iv: "2e48193aa9bf120ff52dccb7a7f9a638"
server_key_version: "encryptionkey@0"
I20260808 15:49:27.670086 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "a5fcb63d23c34a63b7dae6e202fad886"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "13998a43a38806e006d61a5c47b5ba40"
server_key_iv: "2e48193aa9bf120ff52dccb7a7f9a638"
server_key_version: "encryptionkey@0"
I20260808 15:49:27.675462 8587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:49:27.679033 8605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:27.680516 8587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:49:27.680665 8587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "a5fcb63d23c34a63b7dae6e202fad886"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "13998a43a38806e006d61a5c47b5ba40"
server_key_iv: "2e48193aa9bf120ff52dccb7a7f9a638"
server_key_version: "encryptionkey@0"
I20260808 15:49:27.680873 8587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:27.718112 8587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:27.723239 8587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:27.723522 8587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:27.724548 8587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:27.726011 8587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:27.726099 8587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:27.726198 8587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:27.726262 8587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:27.761397 8587 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:41283
I20260808 15:49:27.761445 8787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:41283 every 8 connection(s)
I20260808 15:49:27.762797 8587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:27.770977 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 8587
I20260808 15:49:27.771201 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:27.771899 27987 external_mini_cluster.cc:1428] Setting key 39b3a06989a22cca2cfc30766d9f906a
Aug 08 15:49:27 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204167, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:27.784222 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:27.765586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:33103 (local address 127.27.84.254:44445)
0808 15:49:27.766146 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:27.766151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:27.767005 (+ 854us) server_negotiation.cc:407] Connection header received
0808 15:49:27.768945 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:27.768952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:27.769047 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:27.769174 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:27.770832 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:27.771709 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:27.773620 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:27.774170 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:27.774318 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:27.778291 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:27.778297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:27.778300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:27.778335 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:27.780927 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:27.781717 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:27.781721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:27.781723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:27.781846 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:27.782328 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:27.782333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:27.782335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:27.782646 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:27.782816 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:27.783576 (+ 760us) server_negotiation.cc:299] Negotiation successful
0808 15:49:27.783825 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":316,"threads_started":1}
I20260808 15:49:27.785408 8796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:27.766042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:33103)
0808 15:49:27.766833 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:27.766878 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:27.768658 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:27.769438 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:27.769447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:27.769983 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:27.770575 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:27.770594 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:27.771896 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:27.771900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:27.773461 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:27.773475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:27.774425 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:27.774434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:27.774745 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:27.775501 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:27.775525 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:27.778037 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:27.781079 (+ 3042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:27.781088 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:27.781108 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:27.781564 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:27.781966 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:27.781971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:27.781975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:27.782185 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:27.782792 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:27.782800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:27.783170 (+ 370us) client_negotiation.cc:769] Sending connection context
0808 15:49:27.783469 (+ 299us) client_negotiation.cc:240] Negotiation successful
0808 15:49:27.784039 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":300,"threads_started":1}
I20260808 15:49:27.787171 8788 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44445
I20260808 15:49:27.787529 8788 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:27.788436 8788 heartbeater.cc:511] Master 127.27.84.254:44445 requested a full tablet report, sending...
I20260808 15:49:27.790593 8490 ts_manager.cc:194] Registered new tserver with Master: a5fcb63d23c34a63b7dae6e202fad886 (127.27.84.193:41283)
I20260808 15:49:27.792182 8490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:33103
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:49:27.842149 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44445
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:28.198484 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:28.198880 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:28.199079 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:28.206468 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:28.206569 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:28.206760 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:28.216596 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:28.217953 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:28.219286 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:28.233559 8808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:28.233642 8809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:28.233604 8813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:28.235031 8800 server_base.cc:1034] running on GCE node
I20260808 15:49:28.235736 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:28.236807 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:28.238046 8800 hybrid_clock.cc:648] HybridClock initialized: now 1786204168237977 us; error 68 us; skew 500 ppm
I20260808 15:49:28.238399 8800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:28 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:28.247295 8800 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:49:28.249003 8800 webserver.cc:460] Webserver started at http://127.27.84.194:42797/ using document root <none> and password file <none>
I20260808 15:49:28.249787 8800 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:28.249941 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:28.250272 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:28.253401 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "d54d0a3ab1804db9bf3f1792f5616d59"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "b4cb8c3f71d414301c92d177d8502408"
server_key_iv: "e203344615548ae40619fe99fd3eaf88"
server_key_version: "encryptionkey@0"
I20260808 15:49:28.254221 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "d54d0a3ab1804db9bf3f1792f5616d59"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "b4cb8c3f71d414301c92d177d8502408"
server_key_iv: "e203344615548ae40619fe99fd3eaf88"
server_key_version: "encryptionkey@0"
I20260808 15:49:28.259658 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:49:28.263119 8822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:28.264663 8800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260808 15:49:28.264858 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "d54d0a3ab1804db9bf3f1792f5616d59"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "b4cb8c3f71d414301c92d177d8502408"
server_key_iv: "e203344615548ae40619fe99fd3eaf88"
server_key_version: "encryptionkey@0"
I20260808 15:49:28.265039 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:28.286791 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:28.291455 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:28.291757 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:28.292812 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:28.294216 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:28.294297 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:28.294454 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:28.294507 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:28.330657 8800 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:38611
I20260808 15:49:28.330775 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:38611 every 8 connection(s)
I20260808 15:49:28.332043 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:28.333623 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 8800
I20260808 15:49:28.333817 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:28.334280 27987 external_mini_cluster.cc:1428] Setting key 9ee1a6155bfe3e1a36b8fb5df27a0e22
Aug 08 15:49:28 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:28.353596 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:28.335008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:56041 (local address 127.27.84.254:44445)
0808 15:49:28.335526 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:28.335532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:28.336777 (+ 1245us) server_negotiation.cc:407] Connection header received
0808 15:49:28.338700 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:28.338706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:28.338793 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:28.338930 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:28.340499 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.341236 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.342985 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.343483 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.343624 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:28.347248 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:28.347253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:28.347256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:28.347290 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:28.350035 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:28.351079 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:28.351083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:28.351086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:28.351232 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:28.351776 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:28.351780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:28.351782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:28.352065 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:28.352190 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:28.353132 (+ 942us) server_negotiation.cc:299] Negotiation successful
0808 15:49:28.353322 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260808 15:49:28.355368 9013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:28.335699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:56041)
0808 15:49:28.336608 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:28.336655 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:28.338406 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:28.339199 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:28.339207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:28.339725 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:28.340313 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:28.340331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.341420 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.341424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:28.342835 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:28.342849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.343728 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.343736 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:28.343985 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:28.344697 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:28.344725 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:28.347016 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:28.350205 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:28.350217 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:28.350237 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:28.350934 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:28.351354 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:28.351362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:28.351369 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:28.351615 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:28.352204 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:28.352215 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:28.352671 (+ 456us) client_negotiation.cc:769] Sending connection context
0808 15:49:28.353018 (+ 347us) client_negotiation.cc:240] Negotiation successful
0808 15:49:28.353646 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":361,"threads_started":1}
I20260808 15:49:28.356979 9010 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44445
I20260808 15:49:28.357307 9010 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:28.358048 9010 heartbeater.cc:511] Master 127.27.84.254:44445 requested a full tablet report, sending...
I20260808 15:49:28.359629 8490 ts_manager.cc:194] Registered new tserver with Master: d54d0a3ab1804db9bf3f1792f5616d59 (127.27.84.194:38611)
I20260808 15:49:28.360447 8490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:56041
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:49:28.404223 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44445
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:28.753456 9017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:28.753810 9017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:28.754014 9017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:28.762115 9017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:28.762272 9017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:28.762465 9017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:28.772910 9017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:28.774333 9017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:28.775774 9017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:28.790031 9023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:28.789950 9028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:28.789986 9024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:28.792192 9017 server_base.cc:1034] running on GCE node
I20260808 15:49:28.792915 9017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:28.794020 9017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:28.795260 9017 hybrid_clock.cc:648] HybridClock initialized: now 1786204168795203 us; error 32 us; skew 500 ppm
I20260808 15:49:28.795352 8788 heartbeater.cc:503] Master 127.27.84.254:44445 was elected leader, sending a full tablet report...
I20260808 15:49:28.795655 9017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:28 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:28.799242 9017 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:49:28.800928 9017 webserver.cc:460] Webserver started at http://127.27.84.195:42035/ using document root <none> and password file <none>
I20260808 15:49:28.801669 9017 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:28.801832 9017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:28.802170 9017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:28.805565 9017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "8119905ec6314c50a92ac207b042f3c1"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "0fcbf06fb88b451949d2d33f96b79556"
server_key_iv: "0de84458e0bbeb7e22627b411aaa2e9e"
server_key_version: "encryptionkey@0"
I20260808 15:49:28.806391 9017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "8119905ec6314c50a92ac207b042f3c1"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "0fcbf06fb88b451949d2d33f96b79556"
server_key_iv: "0de84458e0bbeb7e22627b411aaa2e9e"
server_key_version: "encryptionkey@0"
I20260808 15:49:28.811857 9017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:49:28.815687 9038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:28.817055 9017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:49:28.817235 9017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "8119905ec6314c50a92ac207b042f3c1"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "0fcbf06fb88b451949d2d33f96b79556"
server_key_iv: "0de84458e0bbeb7e22627b411aaa2e9e"
server_key_version: "encryptionkey@0"
I20260808 15:49:28.817446 9017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:28.833475 9017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:28.837994 9017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:28.838342 9017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:28.839473 9017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:28.840931 9017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:28.841039 9017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:28.841199 9017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:28.841279 9017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:28.876878 9017 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:34891
I20260808 15:49:28.876936 9209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:34891 every 8 connection(s)
I20260808 15:49:28.878268 9017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:28.885280 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 9017
I20260808 15:49:28.885491 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:28.886163 27987 external_mini_cluster.cc:1428] Setting key 25e1da4592a16f3363f8f915bc9dbf7c
Aug 08 15:49:28 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204168, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:28.901863 9216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:28.881126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:48623 (local address 127.27.84.254:44445)
0808 15:49:28.881584 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:28.881588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:28.882722 (+ 1134us) server_negotiation.cc:407] Connection header received
0808 15:49:28.885606 (+ 2884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:28.885614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:28.885720 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:28.885811 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:28.888404 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.889342 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.891088 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.891501 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.891599 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:28.895302 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:28.895307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:28.895310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:28.895350 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:28.898045 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:28.898904 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:28.898909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:28.898912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:28.899037 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:28.899564 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:28.899581 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:28.899603 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:28.900084 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:28.900244 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:28.901212 (+ 968us) server_negotiation.cc:299] Negotiation successful
0808 15:49:28.901435 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260808 15:49:28.903048 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:28.881762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:48623)
0808 15:49:28.882517 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:28.882557 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:28.885197 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:28.886495 (+ 1298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:28.886507 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:28.887365 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:28.888145 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:28.888189 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.889531 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.889537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:28.890937 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:28.890950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:28.891693 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:28.891698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:28.891960 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:28.892617 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:28.892640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:28.895066 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:28.898219 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:28.898229 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:28.898247 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:28.898771 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:28.899153 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:28.899158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:28.899162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:28.899406 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:28.900221 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:28.900230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:28.900722 (+ 492us) client_negotiation.cc:769] Sending connection context
0808 15:49:28.901071 (+ 349us) client_negotiation.cc:240] Negotiation successful
0808 15:49:28.901663 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":292,"threads_started":1}
I20260808 15:49:28.904765 9212 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44445
I20260808 15:49:28.905071 9212 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:28.905954 9212 heartbeater.cc:511] Master 127.27.84.254:44445 requested a full tablet report, sending...
I20260808 15:49:28.907505 8490 ts_manager.cc:194] Registered new tserver with Master: 8119905ec6314c50a92ac207b042f3c1 (127.27.84.195:34891)
I20260808 15:49:28.908350 8490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:48623
I20260808 15:49:28.914983 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab.
Entry for principal oryx/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab.
WARNING: no policy specified for oryx/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab.
Entry for principal oryx/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab.
WARNING: no policy specified for oryx/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab.
Entry for principal oryx/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab.
I20260808 15:49:29.106135 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 8587
Aug 08 15:49:29 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:49:29.125428 9227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:29.107457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.27.84.193:41283)
0808 15:49:29.108195 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:29.108202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:29.108231 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:49:29.108298 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:29.108305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:29.108550 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:29.108693 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:29.110162 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:29.111388 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:29.112842 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:29.113456 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:29.113568 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:29.118357 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:29.118374 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:29.118392 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:29.118443 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:29.122379 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:29.123303 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:29.123312 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:29.123314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:29.123456 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:29.123834 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:29.123838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:29.123840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:29.124232 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:29.124494 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:29.124853 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 15:49:29.125093 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":387,"threads_started":1}
I20260808 15:49:29.126907 8782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:29.211146 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 8587
I20260808 15:49:29.236351 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 8800
Aug 08 15:49:29 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:49:29.252970 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:29.237396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.27.84.194:38611)
0808 15:49:29.238031 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:29.238039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:29.238065 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:49:29.238127 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:29.238133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:29.238341 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:29.238478 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:29.239819 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:29.240847 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:29.242081 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:29.242712 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:29.242825 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:29.247118 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:29.247130 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:29.247145 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:29.247186 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:29.250033 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:29.250748 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:29.250756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:29.250758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:29.250926 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:29.251293 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:29.251297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:29.251299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:29.251657 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:29.251930 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:29.252216 (+ 286us) server_negotiation.cc:299] Negotiation successful
0808 15:49:29.252585 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":311,"threads_started":1}
I20260808 15:49:29.254362 9002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:29.339514 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 8800
I20260808 15:49:29.365571 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9017
Aug 08 15:49:29 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:49:29.384014 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:29.366875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.27.84.195:34891)
0808 15:49:29.367513 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:29.367521 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:29.367555 (+ 34us) server_negotiation.cc:407] Connection header received
0808 15:49:29.367622 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:29.367629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:29.367914 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:29.368074 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:29.369386 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:29.370537 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:29.372132 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:29.372731 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:29.372848 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:29.377792 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:29.377805 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:29.377821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:29.377856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:29.380837 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:29.381589 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:29.381598 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:29.381601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:29.381763 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:29.382230 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:29.382235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:29.382237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:29.382840 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:29.383090 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:29.383449 (+ 359us) server_negotiation.cc:299] Negotiation successful
0808 15:49:29.383684 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":300,"threads_started":1}
I20260808 15:49:29.385555 9204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:29.471738 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 9017
I20260808 15:49:29.495560 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 8447
I20260808 15:49:29.496961 8540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:29.604008 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 8447
I20260808 15:49:29.632635 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44445
--webserver_interface=127.27.84.254
--webserver_port=44767
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:44445
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:29.981045 9240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:29.981411 9240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:29.981585 9240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:29.988988 9240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:29.989142 9240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:29.989251 9240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:29.989360 9240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:29.989449 9240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:29.999387 9240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:44445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=44767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:30.000829 9240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:30.002282 9240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:30.015619 9249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:30.015794 9252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:30.015848 9250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:30.016692 9240 server_base.cc:1034] running on GCE node
I20260808 15:49:30.017740 9240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:30.019505 9240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:30.020831 9240 hybrid_clock.cc:648] HybridClock initialized: now 1786204170020766 us; error 60 us; skew 500 ppm
I20260808 15:49:30.021258 9240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:30 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204170, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:30.024955 9240 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:49:30.026592 9240 webserver.cc:460] Webserver started at http://127.27.84.254:44767/ using document root <none> and password file <none>
I20260808 15:49:30.027467 9240 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:30.027647 9240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:30.033890 9240 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260808 15:49:30.037001 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:30.038522 9240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:30.038746 9240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "7eb77613fd8f468f81f007c0460b2ada"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "9581e38b144a52bd6a82dc299df48a71"
server_key_iv: "7f22c4db7ac398675f8b4a086b2d6dd5"
server_key_version: "encryptionkey@0"
I20260808 15:49:30.039788 9240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:30.060604 9240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:30.065284 9240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:30.065623 9240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:30.086372 9240 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:44445
I20260808 15:49:30.086403 9339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:44445 every 8 connection(s)
I20260808 15:49:30.088145 9240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:30.092602 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 9240
I20260808 15:49:30.093559 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:41283
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:44445
--webserver_port=33979
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:49:30.095810 9341 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:49:30.099277 9341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Bootstrap starting.
I20260808 15:49:30.122792 9341 log.cc:824] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:30.135464 9341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:49:30.136125 9341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Bootstrap complete.
I20260808 15:49:30.149920 9341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:30.151150 9341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb77613fd8f468f81f007c0460b2ada, State: Initialized, Role: FOLLOWER
I20260808 15:49:30.151924 9341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:30.152227 9341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:30.152364 9341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:30.152541 9341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:49:30.158139 9341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:30.158808 9341 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb77613fd8f468f81f007c0460b2ada; no voters:
I20260808 15:49:30.159499 9341 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:49:30.159652 9346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:49:30.160549 9346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [term 2 LEADER]: Becoming Leader. State: Replica: 7eb77613fd8f468f81f007c0460b2ada, State: Running, Role: LEADER
I20260808 15:49:30.161165 9346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } }
I20260808 15:49:30.161942 9341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:30.163764 9348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb77613fd8f468f81f007c0460b2ada. Latest consensus state: current_term: 2 leader_uuid: "7eb77613fd8f468f81f007c0460b2ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } } }
I20260808 15:49:30.164145 9348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:30.166725 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7eb77613fd8f468f81f007c0460b2ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb77613fd8f468f81f007c0460b2ada" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44445 } } }
I20260808 15:49:30.167121 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:30.168629 9355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:30.174991 9355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:30.176509 9355 catalog_manager.cc:1259] Loaded cluster ID: 5d0a6bb118c644b68d96acb2a7d9ac09
I20260808 15:49:30.176682 9355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:30.180876 9355 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:30.182570 9355 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7eb77613fd8f468f81f007c0460b2ada: Loaded TSK: 0
I20260808 15:49:30.184316 9355 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:49:30.474265 9344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:30.474546 9344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:30.474741 9344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:30.481941 9344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:30.482048 9344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:30.482220 9344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:30.492010 9344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.193.keytab
--principal=oryx/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:41283
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=33979
--tserver_master_addrs=127.27.84.254:44445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:30.493368 9344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:30.494825 9344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:30.508916 9381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:30.508884 9382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:30.509864 9385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:30.510314 9344 server_base.cc:1034] running on GCE node
I20260808 15:49:30.511061 9344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:30.512094 9344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:30.513332 9344 hybrid_clock.cc:648] HybridClock initialized: now 1786204170513268 us; error 56 us; skew 500 ppm
I20260808 15:49:30.513674 9344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:30 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204170, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:30.517122 9344 init.cc:377] Logged in from keytab as oryx/127.27.84.193@KRBTEST.COM (short username oryx)
I20260808 15:49:30.518837 9344 webserver.cc:460] Webserver started at http://127.27.84.193:33979/ using document root <none> and password file <none>
I20260808 15:49:30.519582 9344 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:30.519752 9344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:30.525396 9344 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 15:49:30.528316 9394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:30.529776 9344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:49:30.529978 9344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "a5fcb63d23c34a63b7dae6e202fad886"
format_stamp: "Formatted at 2026-08-08 15:49:27 on dist-test-slave-twwt"
server_key: "13998a43a38806e006d61a5c47b5ba40"
server_key_iv: "2e48193aa9bf120ff52dccb7a7f9a638"
server_key_version: "encryptionkey@0"
I20260808 15:49:30.531091 9344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:30.561455 9344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:30.566890 9344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:30.567227 9344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:30.568357 9344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:30.569998 9344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:30.570099 9344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:30.570209 9344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:30.570314 9344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:30.604635 9344 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:41283
I20260808 15:49:30.604741 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:41283 every 8 connection(s)
I20260808 15:49:30.606256 9344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:30.611073 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 9344
I20260808 15:49:30.612059 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:38611
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:44445
--webserver_port=42797
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:49:30 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:30 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:49:31.079727 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:31.080178 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:31.080435 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:31.096731 9582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:31.096908 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:31.097227 9582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:31.118628 9582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.194.keytab
--principal=oryx/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:38611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=42797
--tserver_master_addrs=127.27.84.254:44445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:31.120666 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:31.122574 9582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:31.146643 9595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:31.146955 9593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:31.156437 9597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:31.156960 9582 server_base.cc:1034] running on GCE node
I20260808 15:49:31.158329 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:31.159897 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:31.161448 9582 hybrid_clock.cc:648] HybridClock initialized: now 1786204171161337 us; error 96 us; skew 500 ppm
I20260808 15:49:31.161939 9582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204171, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:31.167527 9582 init.cc:377] Logged in from keytab as oryx/127.27.84.194@KRBTEST.COM (short username oryx)
I20260808 15:49:31.170148 9582 webserver.cc:460] Webserver started at http://127.27.84.194:42797/ using document root <none> and password file <none>
I20260808 15:49:31.171291 9582 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:31.171480 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:31.180778 9582 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.001s
I20260808 15:49:31.185341 9611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:31.187856 9582 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260808 15:49:31.188120 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "d54d0a3ab1804db9bf3f1792f5616d59"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "b4cb8c3f71d414301c92d177d8502408"
server_key_iv: "e203344615548ae40619fe99fd3eaf88"
server_key_version: "encryptionkey@0"
I20260808 15:49:31.189808 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:31.219094 9582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:31.223958 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:31.224270 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:31.225497 9582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:31.227133 9582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:31.227264 9582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:31.227414 9582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:31.227507 9582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:31.278308 9582 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:38611
I20260808 15:49:31.278367 9787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:38611 every 8 connection(s)
I20260808 15:49:31.279983 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:31.290323 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 9582
I20260808 15:49:31.291486 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:34891
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:44445
--webserver_port=42035
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:49:31.882083 9799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:31.882432 9799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:31.882648 9799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:31.890031 9799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:31.890199 9799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:31.890396 9799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:31.902240 9799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/krb5kdc/oryx_127.27.84.195.keytab
--principal=oryx/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:34891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=42035
--tserver_master_addrs=127.27.84.254:44445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:31.903769 9799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:31.905186 9799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:31.909557 9580 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.288s user 0.003s sys 0.013s
I20260808 15:49:31.911763 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:30.609794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:57921)
0808 15:49:30.611085 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:30.611136 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:30.613100 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:30.614659 (+ 1559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:30.614672 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:30.615351 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:30.615941 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:30.615961 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:30.617650 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:30.617654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:30.619038 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:30.619051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:30.620169 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:30.620177 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:30.620479 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:30.621223 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:30.621248 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:31.910582 (+1289334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":981,"thread_start_us":783,"threads_started":1}
W20260808 15:49:31.912762 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:30.609825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:57921 (local address 127.27.84.254:44445)
0808 15:49:30.611985 (+ 2160us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:30.612001 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:30.612059 (+ 58us) server_negotiation.cc:407] Connection header received
0808 15:49:30.613706 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:30.613720 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:30.614079 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:30.614470 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:30.616174 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:30.617451 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:30.619193 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:30.619905 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:30.620140 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:31.911352 (+1291212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:57921: BlockingRecv error: recv got EOF from 127.27.84.193:57921 (error 108)
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":1716,"threads_started":1}
W20260808 15:49:31.913684 9574 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:44445 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.84.254:44445: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
W20260808 15:49:31.927778 9807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:49:31.933307 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:31.914857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:56187 (local address 127.27.84.254:44445)
0808 15:49:31.915035 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:31.915040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:31.915806 (+ 766us) server_negotiation.cc:407] Connection header received
0808 15:49:31.915928 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:31.915937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:31.916062 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:31.916229 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:31.917936 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.919636 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.922576 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.923585 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.923761 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:31.932931 (+ 9170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:56187: BlockingRecv error: recv got EOF from 127.27.84.193:56187 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:49:31.934109 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:31.915164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:56187)
0808 15:49:31.915572 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:31.915599 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:31.915766 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:31.916536 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:31.916545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:31.917278 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:31.917618 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:31.917637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.919927 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.919936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:31.922304 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:31.922329 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.924153 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.924164 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:31.924446 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:31.925330 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:31.925368 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:31.933666 (+ 8298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231}
W20260808 15:49:31.934465 9812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:31.934653 9809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:31.936257 9799 server_base.cc:1034] running on GCE node
I20260808 15:49:31.937326 9799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:31.938774 9799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:31.942193 9799 hybrid_clock.cc:648] HybridClock initialized: now 1786204171942087 us; error 85 us; skew 500 ppm
I20260808 15:49:31.942694 9799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:31.946326 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:31.935173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:42573)
0808 15:49:31.935397 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:31.935412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:31.935497 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:31.936190 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:31.936199 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:31.936831 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:31.937140 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:31.937157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.938673 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.938679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:31.939822 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:31.939835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.940620 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.940627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:31.940821 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:31.941544 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:31.941561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:31.946010 (+ 4449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
Aug 08 15:49:31 dist-test-slave-twwt krb5kdc[8421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204171, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:49:31.946765 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:31.935521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:42573 (local address 127.27.84.254:44445)
0808 15:49:31.935765 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:31.935771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:31.935791 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:49:31.935877 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:31.935883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:31.935978 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:31.936085 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:31.937324 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.938422 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.939990 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.940448 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.940586 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:31.946382 (+ 5796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:42573: BlockingRecv error: recv got EOF from 127.27.84.193:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260808 15:49:31.946877 9574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:49:31.947415 9799 init.cc:377] Logged in from keytab as oryx/127.27.84.195@KRBTEST.COM (short username oryx)
I20260808 15:49:31.949223 9799 webserver.cc:460] Webserver started at http://127.27.84.195:42035/ using document root <none> and password file <none>
I20260808 15:49:31.950074 9799 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:31.950243 9799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:31.960204 9799 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.001s
I20260808 15:49:31.964560 9826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:31.966686 9799 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260808 15:49:31.966917 9799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "8119905ec6314c50a92ac207b042f3c1"
format_stamp: "Formatted at 2026-08-08 15:49:28 on dist-test-slave-twwt"
server_key: "0fcbf06fb88b451949d2d33f96b79556"
server_key_iv: "0de84458e0bbeb7e22627b411aaa2e9e"
server_key_version: "encryptionkey@0"
I20260808 15:49:31.968014 9799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:32.007377 9799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:32.012265 9799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:32.012576 9799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:32.013859 9799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:32.015882 9799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:32.016018 9799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:32.016165 9799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:32.016258 9799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:32.066247 9799 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:34891
I20260808 15:49:32.066295 10021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:34891 every 8 connection(s)
I20260808 15:49:32.068184 9799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:32.075815 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 9799
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:32.113925 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:32.076874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34040 (local address 127.27.84.254:44445)
0808 15:49:32.092078 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:32.092084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:32.092103 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:49:32.092257 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:32.092262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:32.092355 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:32.092467 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:32.094197 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.096133 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.098393 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.099424 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.099620 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.103654 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:32.103671 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:32.103711 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:32.103781 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:32.109869 (+ 6088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:32.110955 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:32.110964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:32.110972 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:32.111175 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:32.111625 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:32.111629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:32.111631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:32.112164 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:32.112526 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:32.113079 (+ 553us) server_negotiation.cc:299] Negotiation successful
0808 15:49:32.113503 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15062,"thread_start_us":15226,"threads_started":1}
W20260808 15:49:32.578059 9794 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.284s user 0.004s sys 0.011s
I20260808 15:49:32.579986 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:31.284406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:47091)
0808 15:49:31.285407 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:31.285457 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:31.287355 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:31.288126 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:31.288134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:31.288661 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:31.289227 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:31.289244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.290408 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.290412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:31.291880 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:31.291896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.292665 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.292675 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:31.292951 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:31.293638 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:31.293666 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:32.579083 (+1285417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":437,"threads_started":1}
W20260808 15:49:32.580034 9793 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:31.283328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:47091 (local address 127.27.84.254:44445)
0808 15:49:31.283850 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:31.283855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:31.285591 (+ 1736us) server_negotiation.cc:407] Connection header received
0808 15:49:31.287629 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:31.287635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:31.287730 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:31.287863 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:31.289417 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.290234 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.292019 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:31.292418 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:31.292566 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.579467 (+1286901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:47091: BlockingRecv error: recv got EOF from 127.27.84.194:47091 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":256,"threads_started":1}
W20260808 15:49:32.581163 9791 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:44445 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.84.254:44445: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:32.591295 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:32.581969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:54997)
0808 15:49:32.582212 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:32.582235 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:32.582360 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:32.582745 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:32.582750 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:32.583155 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:32.583321 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.583330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.584375 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.584383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.585343 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.585354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.586076 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.586083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.586255 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.586950 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:32.586969 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:32.590972 (+ 4003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88}
W20260808 15:49:32.591441 9793 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:32.582132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:54997 (local address 127.27.84.254:44445)
0808 15:49:32.582312 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:32.582318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:32.582348 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:49:32.582504 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:32.582511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:32.582578 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:32.582716 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:32.583467 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.584222 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.585482 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.585894 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.586034 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.591180 (+ 5146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:54997: BlockingRecv error: recv got EOF from 127.27.84.194:54997 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:32.601686 9794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:32.592439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:52409)
0808 15:49:32.592617 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:32.592637 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:32.592760 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:32.593104 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:32.593109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:32.593446 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:32.593606 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.593614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.594673 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.594679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.596252 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.596266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.597007 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.597012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.597222 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.597851 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:32.597869 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:32.601386 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:49:32.601918 9793 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:32.592543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:52409 (local address 127.27.84.254:44445)
0808 15:49:32.592723 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:32.592728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:32.592746 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:49:32.592867 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:32.592873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:32.592963 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:32.593068 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:32.593750 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.594500 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.596405 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.596829 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.596965 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.601619 (+ 4654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:52409: BlockingRecv error: recv got EOF from 127.27.84.194:52409 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 15:49:32.602349 9791 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:32 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:32.959622 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:32.947478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:55033)
0808 15:49:32.948065 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:32.948083 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:32.948200 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:32.948676 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:32.948679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:32.949112 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:32.949266 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.949274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.951356 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.951360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.952440 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.952455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.953847 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.953852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.953979 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.954695 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:32.954714 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:32.959373 (+ 4659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":345,"threads_started":1}
W20260808 15:49:32.960132 9793 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:32.947614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55033 (local address 127.27.84.254:44445)
0808 15:49:32.947880 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:32.947889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:32.948172 (+ 283us) server_negotiation.cc:407] Connection header received
0808 15:49:32.948341 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:32.948349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:32.948489 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:32.948638 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:32.949436 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.951177 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.952601 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.953636 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.953819 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.959757 (+ 5938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:55033: BlockingRecv error: recv got EOF from 127.27.84.193:55033 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260808 15:49:33.228080 10026 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.136s user 0.010s sys 0.009s
W20260808 15:49:33.229300 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:32.072197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:60133 (local address 127.27.84.254:44445)
0808 15:49:32.072429 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:32.072435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:32.074280 (+ 1845us) server_negotiation.cc:407] Connection header received
0808 15:49:32.078625 (+ 4345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:32.078636 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:32.078769 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:32.078882 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:32.082477 (+ 3595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.084134 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.087947 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.089057 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.089204 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.229095 (+1139891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:60133: BlockingRecv error: recv got EOF from 127.27.84.195:60133 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:49:33.229912 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:32.072835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:60133)
0808 15:49:32.074034 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:32.074106 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:32.078049 (+ 3943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:32.079720 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:32.079738 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:32.080923 (+ 1185us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:32.082144 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.082183 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.084536 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.084546 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.087674 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:32.087702 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:32.089774 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:32.089786 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:32.090267 (+ 481us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:32.091540 (+ 1273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:32.091598 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:33.228902 (+1137304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":425,"threads_started":1}
W20260808 15:49:33.230980 10022 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:44445 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.84.254:44445: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:33.241551 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:33.231637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:47251)
0808 15:49:33.231843 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:33.231882 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:33.232006 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:33.232347 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:33.232351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:33.232743 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:33.232912 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.232921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.234011 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.234017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.235106 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.235118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.235919 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.235925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.236097 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.236851 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:33.236874 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:33.241224 (+ 4350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 15:49:33.241709 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:33.231776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:47251 (local address 127.27.84.254:44445)
0808 15:49:33.232014 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:33.232020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:33.232040 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:49:33.232136 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:33.232141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:33.232201 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:33.232309 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:33.233063 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.233862 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.235247 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.235727 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.235890 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.241443 (+ 5553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:47251: BlockingRecv error: recv got EOF from 127.27.84.195:47251 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:33.257980 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:33.242373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:57255)
0808 15:49:33.242589 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:33.242650 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:33.242765 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:33.243133 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:33.243137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:33.243523 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:33.243743 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.243754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.244742 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.244746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.245702 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.245714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.246528 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.246532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.246760 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.254060 (+ 7300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:33.254082 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:33.257727 (+ 3645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260808 15:49:33.258270 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:33.242504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:57255 (local address 127.27.84.254:44445)
0808 15:49:33.242728 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:33.242732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:33.242750 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:49:33.242887 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:33.242891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:33.242952 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:33.243036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:33.243889 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.244615 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.245920 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.246378 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.246487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.257994 (+ 11507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:57255: BlockingRecv error: recv got EOF from 127.27.84.195:57255 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 15:49:33.258342 10022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:33.613534 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:33.602930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:43963)
0808 15:49:33.603556 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:33.603574 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:33.603695 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:33.604033 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:33.604036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:33.604518 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:33.604678 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.604687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.606184 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.606189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.607288 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.607303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.608092 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.608097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.608253 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.608889 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:33.608910 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:33.613168 (+ 4258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":383,"threads_started":1}
W20260808 15:49:33.613690 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:33.603050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:43963 (local address 127.27.84.254:44445)
0808 15:49:33.603255 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:33.603260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:33.603653 (+ 393us) server_negotiation.cc:407] Connection header received
0808 15:49:33.603825 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:33.603828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:33.603894 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:33.603978 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:33.604884 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.606006 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.607458 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.607882 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.607989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.613419 (+ 5430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:43963: BlockingRecv error: recv got EOF from 127.27.84.194:43963 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:33 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:33.970575 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:33.960801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:36345)
0808 15:49:33.961364 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:33.961384 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:33.961559 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:33.961973 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:33.961978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:33.962561 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:33.962862 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.962878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.963968 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.963972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.964878 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:33.964890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.965622 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.965626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:33.965748 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.966345 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:33.966366 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:33.970298 (+ 3932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":305,"threads_started":1}
W20260808 15:49:33.970867 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:33.960879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:36345 (local address 127.27.84.254:44445)
0808 15:49:33.961090 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:33.961097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:33.961507 (+ 410us) server_negotiation.cc:407] Connection header received
0808 15:49:33.961681 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:33.961688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:33.961801 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:33.961912 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:33.963015 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.963845 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.965036 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:33.965478 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:33.965583 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:33.970523 (+ 4940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:36345: BlockingRecv error: recv got EOF from 127.27.84.193:36345 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 08 15:49:34 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:34 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:34.269748 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:34.259104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:38357)
0808 15:49:34.260031 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:34.260053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:34.260211 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:34.260605 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:34.260612 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:34.261127 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:34.261338 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:34.261352 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.262860 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.262864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:34.263953 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:34.263966 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.264956 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.264961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:34.265085 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:34.265742 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:34.265760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:34.269509 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":655,"threads_started":1}
W20260808 15:49:34.270048 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:34.259366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:38357 (local address 127.27.84.254:44445)
0808 15:49:34.259546 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:34.259553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:34.260155 (+ 602us) server_negotiation.cc:407] Connection header received
0808 15:49:34.260347 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:34.260353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:34.260435 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:34.260537 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:34.261470 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.262719 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.264124 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.264770 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.264896 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:34.269731 (+ 4835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:38357: BlockingRecv error: recv got EOF from 127.27.84.195:38357 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 08 15:49:34 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:34 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:34.624234 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:34.614612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:47139)
0808 15:49:34.615220 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:34.615236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:34.615426 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:34.615832 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:34.615837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:34.616296 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:34.616493 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:34.616502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.617575 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.617583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:34.618550 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:34.618562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.619469 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.619475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:34.619656 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:34.620241 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:34.620258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:34.623991 (+ 3733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":346,"threads_started":1}
W20260808 15:49:34.624420 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:34.614763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:47139 (local address 127.27.84.254:44445)
0808 15:49:34.615037 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:34.615046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:34.615341 (+ 295us) server_negotiation.cc:407] Connection header received
0808 15:49:34.615563 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:34.615568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:34.615642 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:34.615750 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:34.616648 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.617417 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.618753 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.619293 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.619433 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:34.624216 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:47139: BlockingRecv error: recv got EOF from 127.27.84.194:47139 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 08 15:49:34 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:34 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:34.981766 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:34.971901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:55393)
0808 15:49:34.972558 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:34.972588 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:34.972763 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:34.973139 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:34.973145 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:34.973549 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:34.973786 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:34.973798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.974933 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.974938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:34.975894 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:34.975906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.976655 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.976662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:34.976883 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:34.977573 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:34.977592 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:34.981454 (+ 3862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":396,"threads_started":1}
W20260808 15:49:34.981956 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:34.972066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55393 (local address 127.27.84.254:44445)
0808 15:49:34.972321 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:34.972326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:34.972670 (+ 344us) server_negotiation.cc:407] Connection header received
0808 15:49:34.972920 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:34.972924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:34.972988 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:34.973096 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:34.973944 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.974767 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.976050 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:34.976495 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:34.976639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:34.981702 (+ 5063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:55393: BlockingRecv error: recv got EOF from 127.27.84.193:55393 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 15:49:35 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:35 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:35.281224 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:35.271234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:42131)
0808 15:49:35.271881 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:35.271898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:35.272070 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:35.272435 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:35.272440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:35.273028 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:35.273225 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:35.273234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.274285 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.274291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:35.275323 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:35.275336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.276064 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.276071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:35.276256 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:35.276921 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:35.276951 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:35.280971 (+ 4020us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":399,"threads_started":1}
W20260808 15:49:35.281446 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:35.271315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:42131 (local address 127.27.84.254:44445)
0808 15:49:35.271520 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:35.271541 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:35.271994 (+ 453us) server_negotiation.cc:407] Connection header received
0808 15:49:35.272197 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:35.272203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:35.272278 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:35.272386 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:35.273375 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.274129 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.275477 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.275900 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.276049 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:35.281216 (+ 5167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:42131: BlockingRecv error: recv got EOF from 127.27.84.195:42131 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 08 15:49:35 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:35 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:35.635485 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:35.625306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:42233)
0808 15:49:35.625924 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:35.625943 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:35.626073 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:35.626443 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:35.626446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:35.626937 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:35.627122 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:35.627132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.628292 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.628297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:35.629235 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:35.629247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.630046 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.630050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:35.630181 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:35.630873 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:35.630891 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:35.635095 (+ 4204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":329,"threads_started":1}
W20260808 15:49:35.635700 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:35.625418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:42233 (local address 127.27.84.254:44445)
0808 15:49:35.625604 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:35.625609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:35.626029 (+ 420us) server_negotiation.cc:407] Connection header received
0808 15:49:35.626177 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:35.626181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:35.626288 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:35.626390 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:35.627292 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.628143 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.629415 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.629885 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.630005 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:35.635432 (+ 5427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:42233: BlockingRecv error: recv got EOF from 127.27.84.194:42233 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 08 15:49:35 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:35 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:35.993213 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:35.983003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:40593)
0808 15:49:35.983542 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:35.983559 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:35.983695 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:35.984076 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:35.984080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:35.984493 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:35.984684 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:35.984693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.985766 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.985770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:35.987054 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:35.987069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.988032 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.988038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:35.988247 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:35.989007 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:35.989026 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:35.992948 (+ 3922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":307,"threads_started":1}
W20260808 15:49:35.993422 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:35.983106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:40593 (local address 127.27.84.254:44445)
0808 15:49:35.983354 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:35.983360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:35.983649 (+ 289us) server_negotiation.cc:407] Connection header received
0808 15:49:35.983802 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:35.983806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:35.983900 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:35.983991 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:35.984854 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.985634 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.987230 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:35.987868 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:35.987968 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:35.993199 (+ 5231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:40593: BlockingRecv error: recv got EOF from 127.27.84.193:40593 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 08 15:49:36 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:36 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:36.291934 10045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:36.282228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:37739)
0808 15:49:36.282824 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:36.282841 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:36.283007 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:36.283366 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:36.283371 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:36.283768 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:36.283941 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:36.283951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.285196 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.285204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:36.286569 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:36.286583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.287379 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.287386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:36.287581 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:36.288153 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:36.288170 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:36.291642 (+ 3472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":315,"threads_started":1}
W20260808 15:49:36.292090 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:36.282361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:37739 (local address 127.27.84.254:44445)
0808 15:49:36.282650 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:36.282656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:36.282937 (+ 281us) server_negotiation.cc:407] Connection header received
0808 15:49:36.283141 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:36.283146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:36.283218 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:36.283327 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:36.284098 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.285043 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.286762 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.287204 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.287348 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:36.291854 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:37739: BlockingRecv error: recv got EOF from 127.27.84.195:37739 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 08 15:49:36 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:36 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:36.646248 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:36.636655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:44983)
0808 15:49:36.637224 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:36.637240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:36.637413 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:36.637764 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:36.637770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:36.638214 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:36.638406 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:36.638415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.639537 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.639543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:36.640558 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:36.640570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.641306 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.641320 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:36.641496 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:36.642200 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:36.642229 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:36.645980 (+ 3751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":304,"threads_started":1}
W20260808 15:49:36.646525 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:36.636719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:44983 (local address 127.27.84.254:44445)
0808 15:49:36.636996 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:36.637005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:36.637341 (+ 336us) server_negotiation.cc:407] Connection header received
0808 15:49:36.637542 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:36.637554 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:36.637616 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:36.637725 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:36.638565 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.639383 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.640712 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.641150 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.641290 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:36.646242 (+ 4952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:44983: BlockingRecv error: recv got EOF from 127.27.84.194:44983 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 08 15:49:36 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:36 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:37.003849 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:36.994248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:38149)
0808 15:49:36.994875 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:36.994893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:36.995065 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:36.995427 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:36.995433 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:36.995915 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:36.996069 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:36.996077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.997136 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.997143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:36.998101 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:36.998112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.998919 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.998924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:36.999112 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:36.999734 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:36.999758 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:37.003522 (+ 3764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":359,"threads_started":1}
W20260808 15:49:37.004009 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:36.994369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:38149 (local address 127.27.84.254:44445)
0808 15:49:36.994673 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:36.994680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:36.994989 (+ 309us) server_negotiation.cc:407] Connection header received
0808 15:49:36.995201 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:36.995207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:36.995282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:36.995393 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:36.996221 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.996995 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.998250 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:36.998765 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:36.998904 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:37.003731 (+ 4827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:38149: BlockingRecv error: recv got EOF from 127.27.84.193:38149 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 08 15:49:37 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:37 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:37.302057 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:37.293005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:44979)
0808 15:49:37.293521 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:37.293539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:37.293651 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:37.293980 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:37.293983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:37.294382 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:37.294559 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:37.294569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.295766 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.295771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:37.296856 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:37.296870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.297523 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.297527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:37.297660 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:37.298257 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:37.298276 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:37.301805 (+ 3529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":296,"threads_started":1}
W20260808 15:49:37.302281 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:37.293100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:44979 (local address 127.27.84.254:44445)
0808 15:49:37.293346 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:37.293352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:37.293618 (+ 266us) server_negotiation.cc:407] Connection header received
0808 15:49:37.293749 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:37.293753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:37.293852 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:37.293937 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:37.294747 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.295641 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.296984 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.297395 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.297492 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:37.302014 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:44979: BlockingRecv error: recv got EOF from 127.27.84.195:44979 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 08 15:49:37 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:37 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:37.656484 10053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:37.647189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:36613)
0808 15:49:37.647789 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:37.647808 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:37.647972 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:37.648349 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:37.648355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:37.648817 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:37.648967 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:37.648976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.650002 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.650008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:37.650981 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:37.650993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.651745 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.651751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:37.651928 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:37.652519 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:37.652537 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:37.656181 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":350,"threads_started":1}
W20260808 15:49:37.656653 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:37.647408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36613 (local address 127.27.84.254:44445)
0808 15:49:37.647656 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:37.647663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:37.647903 (+ 240us) server_negotiation.cc:407] Connection header received
0808 15:49:37.648101 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:37.648106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:37.648180 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:37.648294 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:37.649122 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.649854 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.651138 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:37.651586 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:37.651719 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:37.656413 (+ 4694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:36613: BlockingRecv error: recv got EOF from 127.27.84.194:36613 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 15:49:38 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:38 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:38.015090 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:38.005107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:53039)
0808 15:49:38.005630 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:38.005647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:38.005771 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:38.006081 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:38.006084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:38.006532 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:38.006795 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:38.006806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.007935 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.007940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:38.008895 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:38.008909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.009684 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.009689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:38.009844 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:38.010467 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:38.010485 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:38.014839 (+ 4354us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":311,"threads_started":1}
W20260808 15:49:38.015285 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:38.005166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:53039 (local address 127.27.84.254:44445)
0808 15:49:38.005431 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:38.005440 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:38.005739 (+ 299us) server_negotiation.cc:407] Connection header received
0808 15:49:38.005875 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:38.005879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:38.005943 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:38.006039 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:38.006966 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.007756 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.009052 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.009513 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.009625 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:38.015039 (+ 5414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:53039: BlockingRecv error: recv got EOF from 127.27.84.193:53039 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 15:49:38 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:38 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:38.316102 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:38.303191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:33563)
0808 15:49:38.303780 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:38.303797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:38.303970 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:38.304332 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:38.304340 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:38.304866 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:38.305072 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:38.305082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.306631 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.306640 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:38.307602 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:38.307615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.308293 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.308297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:38.308422 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:38.312042 (+ 3620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:38.312061 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:38.315765 (+ 3704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":313,"threads_started":1}
W20260808 15:49:38.316244 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:38.303311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:33563 (local address 127.27.84.254:44445)
0808 15:49:38.303558 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:38.303565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:38.303889 (+ 324us) server_negotiation.cc:407] Connection header received
0808 15:49:38.304098 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:38.304104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:38.304178 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:38.304286 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:38.305277 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.306445 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.307728 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.308145 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.308244 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:38.315975 (+ 7731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:33563: BlockingRecv error: recv got EOF from 127.27.84.195:33563 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 08 15:49:38 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:38 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:38.667399 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:38.657519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:55011)
0808 15:49:38.658165 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:38.658189 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:38.658313 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:38.658698 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:38.658703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:38.659229 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:38.659471 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:38.659484 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.660912 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.660916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:38.661866 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:38.661879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.662696 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.662701 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:38.662841 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:38.663457 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:38.663475 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:38.667135 (+ 3660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":402,"threads_started":1}
W20260808 15:49:38.667538 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:38.657637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:55011 (local address 127.27.84.254:44445)
0808 15:49:38.657873 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:38.657878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:38.658273 (+ 395us) server_negotiation.cc:407] Connection header received
0808 15:49:38.658420 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:38.658424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:38.658489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:38.658582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:38.659720 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.660742 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.662000 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:38.662497 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:38.662647 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:38.667340 (+ 4693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:55011: BlockingRecv error: recv got EOF from 127.27.84.194:55011 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 08 15:49:39 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:39 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:39.025799 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:39.016154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:41241)
0808 15:49:39.016778 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:39.016795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:39.016946 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:39.017329 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:39.017332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:39.017778 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:39.017992 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:39.018001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.019133 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.019138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:39.020108 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:39.020121 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.020871 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.020876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:39.020990 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:39.021605 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:39.021623 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:39.025473 (+ 3850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":319,"threads_started":1}
W20260808 15:49:39.026077 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:39.016256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:41241 (local address 127.27.84.254:44445)
0808 15:49:39.016535 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:39.016545 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:39.016917 (+ 372us) server_negotiation.cc:407] Connection header received
0808 15:49:39.017111 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:39.017116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:39.017184 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:39.017274 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:39.018133 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.018958 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.020254 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.020697 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.020796 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:39.025759 (+ 4963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:41241: BlockingRecv error: recv got EOF from 127.27.84.193:41241 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 08 15:49:39 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:39 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:39.327188 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:39.317380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:42867)
0808 15:49:39.317927 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:39.317950 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:39.318078 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:39.318413 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:39.318416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:39.318895 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:39.319070 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:39.319079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.320185 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.320189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:39.321362 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:39.321375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.322285 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.322290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:39.322435 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:39.323164 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:39.323183 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:39.326950 (+ 3767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":298,"threads_started":1}
W20260808 15:49:39.327375 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:39.317542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:42867 (local address 127.27.84.254:44445)
0808 15:49:39.317733 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:39.317737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:39.318066 (+ 329us) server_negotiation.cc:407] Connection header received
0808 15:49:39.318186 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:39.318189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:39.318261 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:39.318357 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:39.319242 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.320022 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.321472 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.322112 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.322214 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:39.327161 (+ 4947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:42867: BlockingRecv error: recv got EOF from 127.27.84.195:42867 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 15:49:39 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:39 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:39.678304 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:39.668631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:53155)
0808 15:49:39.669376 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:39.669403 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:39.669545 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:39.669877 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:39.669880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:39.670314 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:39.670543 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:39.670555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.671736 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.671742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:39.672899 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:39.672912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.673650 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.673657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:39.673822 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:39.674427 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:39.674445 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:39.677928 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":344,"threads_started":1}
W20260808 15:49:39.678395 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:39.668733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:53155 (local address 127.27.84.254:44445)
0808 15:49:39.668993 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:39.668998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:39.669481 (+ 483us) server_negotiation.cc:407] Connection header received
0808 15:49:39.669665 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:39.669669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:39.669735 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:39.669851 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:39.670807 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.671573 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.673044 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:39.673467 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:39.673626 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:39.678156 (+ 4530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:53155: BlockingRecv error: recv got EOF from 127.27.84.194:53155 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Aug 08 15:49:40 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:40 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:40.037467 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:40.027027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:52779)
0808 15:49:40.027631 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:40.027649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:40.027794 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:40.028313 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:40.028322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:40.028870 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:40.029119 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:40.029132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.030662 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.030666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:40.031769 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:40.031785 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.032649 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.032654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:40.032845 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:40.033521 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:40.033539 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:40.037145 (+ 3606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":327,"threads_started":1}
W20260808 15:49:40.037919 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:40.027195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:52779 (local address 127.27.84.254:44445)
0808 15:49:40.027424 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:40.027428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:40.027769 (+ 341us) server_negotiation.cc:407] Connection header received
0808 15:49:40.027926 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:40.027932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:40.028022 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:40.028161 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:40.029311 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.030408 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.031977 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.032446 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.032562 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:40.037462 (+ 4900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:52779: BlockingRecv error: recv got EOF from 127.27.84.193:52779 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 15:49:40 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:40 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:40.338385 10069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:40.328186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:38341)
0808 15:49:40.328931 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:40.328949 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:40.329124 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:40.329494 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:40.329499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:40.329982 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:40.330163 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:40.330172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.331309 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.331317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:40.332502 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:40.332522 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.333542 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.333549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:40.333724 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:40.334339 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:40.334356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:40.338037 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":318,"threads_started":1}
W20260808 15:49:40.338502 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:40.328389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:38341 (local address 127.27.84.254:44445)
0808 15:49:40.328633 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:40.328641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:40.329050 (+ 409us) server_negotiation.cc:407] Connection header received
0808 15:49:40.329266 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:40.329270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:40.329339 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:40.329450 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:40.330343 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.331128 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.332663 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.333317 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.333464 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:40.338231 (+ 4767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:38341: BlockingRecv error: recv got EOF from 127.27.84.195:38341 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 08 15:49:40 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:40 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:40.689927 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:40.679534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:35041)
0808 15:49:40.680109 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:40.680125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:40.680288 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:40.680651 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:40.680656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:40.681138 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:40.681341 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:40.681351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.682571 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.682578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:40.683696 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:40.683720 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.684644 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.684650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:40.684932 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:40.685834 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:40.685854 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:40.689614 (+ 3760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":306,"threads_started":1}
W20260808 15:49:40.690078 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:40.679632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:35041 (local address 127.27.84.254:44445)
0808 15:49:40.679900 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:40.679909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:40.680217 (+ 308us) server_negotiation.cc:407] Connection header received
0808 15:49:40.680428 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:40.680438 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:40.680500 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:40.680609 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:40.681512 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.682430 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.683860 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:40.684393 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:40.684547 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:40.689830 (+ 5283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:35041: BlockingRecv error: recv got EOF from 127.27.84.194:35041 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 08 15:49:41 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:41 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:41.048554 10073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:41.038424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:46363)
0808 15:49:41.039029 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:41.039047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:41.039162 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:41.039523 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:41.039529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:41.040033 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:41.040185 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:41.040194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.041304 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.041310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:41.042467 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:41.042504 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.043513 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.043520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:41.043718 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:41.044542 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:41.044565 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:41.048307 (+ 3742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":321,"threads_started":1}
W20260808 15:49:41.048789 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:41.038493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:46363 (local address 127.27.84.254:44445)
0808 15:49:41.038711 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:41.038716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:41.039123 (+ 407us) server_negotiation.cc:407] Connection header received
0808 15:49:41.039258 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:41.039262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:41.039370 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:41.039459 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:41.040349 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.041163 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.042679 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.043318 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.043452 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:41.048524 (+ 5072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:46363: BlockingRecv error: recv got EOF from 127.27.84.193:46363 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 08 15:49:41 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:41 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:41.349453 10075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:41.339568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:53177)
0808 15:49:41.340139 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:41.340155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:41.340346 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:41.340734 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:41.340737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:41.341158 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:41.341346 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:41.341355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.342351 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.342355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:41.343409 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:41.343428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.344635 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.344644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:41.344819 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:41.345569 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:41.345586 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:41.349178 (+ 3592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":297,"threads_started":1}
W20260808 15:49:41.349640 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:41.339661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:53177 (local address 127.27.84.254:44445)
0808 15:49:41.339908 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:41.339915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:41.340273 (+ 358us) server_negotiation.cc:407] Connection header received
0808 15:49:41.340526 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:41.340530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:41.340598 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:41.340688 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:41.341480 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.342197 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.343565 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.344369 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.344496 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:41.349409 (+ 4913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:53177: BlockingRecv error: recv got EOF from 127.27.84.195:53177 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 08 15:49:41 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:41 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:41.700569 10077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:41.691037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:47807)
0808 15:49:41.691557 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:41.691574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:41.691689 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:41.691990 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:41.691994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:41.692414 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:41.692565 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:41.692573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.693565 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.693569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:41.694570 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:41.694583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.695437 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.695442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:41.695573 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:41.696237 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:41.696256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:41.700212 (+ 3956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
W20260808 15:49:41.700693 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:41.691147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:47807 (local address 127.27.84.254:44445)
0808 15:49:41.691465 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:41.691472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:41.691654 (+ 182us) server_negotiation.cc:407] Connection header received
0808 15:49:41.691789 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:41.691792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:41.691859 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:41.691942 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:41.692711 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.693438 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.694786 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:41.695220 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:41.695394 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:41.700425 (+ 5031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:47807: BlockingRecv error: recv got EOF from 127.27.84.194:47807 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 08 15:49:42 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:42 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:42.058892 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:42.049525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:42565)
0808 15:49:42.050070 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:42.050087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:42.050208 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:42.050550 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:42.050555 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:42.051021 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:42.051195 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:42.051204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.052218 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.052222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:42.053228 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:42.053239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.053890 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.053894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:42.054003 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:42.054638 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:42.054658 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:42.058480 (+ 3822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":319,"threads_started":1}
W20260808 15:49:42.058943 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:42.049689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:42565 (local address 127.27.84.254:44445)
0808 15:49:42.049935 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:42.049941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:42.050171 (+ 230us) server_negotiation.cc:407] Connection header received
0808 15:49:42.050312 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:42.050316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:42.050397 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:42.050485 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:42.051346 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.052085 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.053364 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.053757 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.053851 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:42.058699 (+ 4848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:42565: BlockingRecv error: recv got EOF from 127.27.84.193:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 08 15:49:42 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:42 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:42.361689 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:42.350516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:51445)
0808 15:49:42.351234 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:42.351252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:42.351366 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:42.351791 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:42.351803 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:42.352221 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:42.352380 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:42.352389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.354105 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.354110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:42.355193 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:42.355231 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.356449 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.356456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:42.356658 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:42.357498 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:42.357517 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:42.361455 (+ 3938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":422,"threads_started":1}
W20260808 15:49:42.362301 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:42.350729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51445 (local address 127.27.84.254:44445)
0808 15:49:42.351026 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:42.351036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:42.351334 (+ 298us) server_negotiation.cc:407] Connection header received
0808 15:49:42.351471 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:42.351479 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:42.351596 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:42.351727 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:42.352534 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.353931 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.355389 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.356267 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.356424 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:42.361807 (+ 5383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:51445: BlockingRecv error: recv got EOF from 127.27.84.195:51445 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 08 15:49:42 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:42 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:42.710788 10081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:42.701579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:39913)
0808 15:49:42.702109 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:42.702125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:42.702288 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:42.702666 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:42.702671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:42.703115 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:42.703280 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:42.703290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.704396 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.704402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:42.705402 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:42.705414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.706187 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.706193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:42.706366 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:42.706970 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:42.706989 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:42.710432 (+ 3443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":300,"threads_started":1}
W20260808 15:49:42.710929 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:42.701647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:39913 (local address 127.27.84.254:44445)
0808 15:49:42.701900 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:42.701908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:42.702215 (+ 307us) server_negotiation.cc:407] Connection header received
0808 15:49:42.702438 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:42.702442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:42.702501 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:42.702607 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:42.703450 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.704254 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.705553 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:42.706020 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:42.706158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:42.710698 (+ 4540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:39913: BlockingRecv error: recv got EOF from 127.27.84.194:39913 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 08 15:49:43 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:43 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:43.069890 10083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:43.059901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:35731)
0808 15:49:43.060474 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:43.060499 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:43.060662 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:43.061038 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:43.061044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:43.061524 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:43.061827 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:43.061842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.063015 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.063023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:43.063984 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:43.063997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.064742 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.064748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:43.064953 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:43.065628 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:43.065647 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:43.069478 (+ 3831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":319,"threads_started":1}
W20260808 15:49:43.069931 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:43.060112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:35731 (local address 127.27.84.254:44445)
0808 15:49:43.060365 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:43.060373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:43.060580 (+ 207us) server_negotiation.cc:407] Connection header received
0808 15:49:43.060788 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:43.060795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:43.060884 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:43.060998 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:43.061991 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.062875 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.064135 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.064570 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.064706 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:43.069628 (+ 4922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:35731: BlockingRecv error: recv got EOF from 127.27.84.193:35731 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 08 15:49:43 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:43 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
W20260808 15:49:43.376035 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:43.363138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:43023 (local address 127.27.84.254:44445)
0808 15:49:43.363463 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:43.363471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:43.364071 (+ 600us) server_negotiation.cc:407] Connection header received
0808 15:49:43.364417 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:43.364424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:43.364577 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:43.364699 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:43.365891 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.367258 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.368894 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.369696 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.369870 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:43.375655 (+ 5785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:43023: BlockingRecv error: recv got EOF from 127.27.84.195:43023 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
I20260808 15:49:43.376216 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:43.362965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:43023)
0808 15:49:43.363963 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:43.363985 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:43.364192 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:43.364874 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:43.364881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:43.365527 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:43.365717 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:43.365727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.367467 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.367477 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:43.368714 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:43.368748 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.369891 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.369897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:43.370021 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:43.370908 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:43.370937 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:43.375782 (+ 4845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":684,"threads_started":1}
Aug 08 15:49:43 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:43 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:43.721496 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:43.712062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:57201)
0808 15:49:43.712598 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:43.712615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:43.712771 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:43.713120 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:43.713123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:43.713567 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:43.713796 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:43.713806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.714897 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.714902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:43.715849 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:43.715860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.716634 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.716639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:43.716784 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:43.717559 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:43.717588 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:43.721258 (+ 3670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":304,"threads_started":1}
W20260808 15:49:43.721779 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:43.712133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:57201 (local address 127.27.84.254:44445)
0808 15:49:43.712379 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:43.712388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:43.712738 (+ 350us) server_negotiation.cc:407] Connection header received
0808 15:49:43.712889 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:43.712894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:43.712960 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:43.713050 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:43.713956 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.714745 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.715992 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:43.716448 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:43.716552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:43.721462 (+ 4910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:57201: BlockingRecv error: recv got EOF from 127.27.84.194:57201 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 08 15:49:44 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:44 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:44.080629 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:44.070894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:38047)
0808 15:49:44.071536 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:44.071553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:44.071668 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:44.072064 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:44.072067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:44.072519 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:44.072714 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:44.072724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.073912 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.073917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:44.074996 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:44.075010 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.075692 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.075697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:44.075861 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:44.076439 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:44.076457 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:44.080269 (+ 3812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":394,"threads_started":1}
W20260808 15:49:44.080768 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:44.070944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:38047 (local address 127.27.84.254:44445)
0808 15:49:44.071188 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:44.071193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:44.071630 (+ 437us) server_negotiation.cc:407] Connection header received
0808 15:49:44.071799 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:44.071803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:44.071929 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:44.072021 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:44.072884 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.073768 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.075136 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.075565 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.075662 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:44.080475 (+ 4813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:38047: BlockingRecv error: recv got EOF from 127.27.84.193:38047 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 08 15:49:44 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:44 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:44.386906 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:44.377229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:37335)
0808 15:49:44.377838 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:44.377856 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:44.378027 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:44.378407 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:44.378411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:44.378967 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:44.379172 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:44.379182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.380283 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.380288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:44.381237 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:44.381250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.382031 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.382036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:44.382189 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:44.382926 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:44.382944 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:44.386596 (+ 3652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":324,"threads_started":1}
W20260808 15:49:44.387301 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:44.377399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:37335 (local address 127.27.84.254:44445)
0808 15:49:44.377568 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:44.377572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:44.377956 (+ 384us) server_negotiation.cc:407] Connection header received
0808 15:49:44.378177 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:44.378181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:44.378266 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:44.378356 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:44.379340 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.380123 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.381396 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.381867 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.381981 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:44.387034 (+ 5053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:37335: BlockingRecv error: recv got EOF from 127.27.84.195:37335 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 08 15:49:44 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:44 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:44.732717 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:44.722807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:53589)
0808 15:49:44.723493 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:44.723512 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:44.723632 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:44.724085 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:44.724091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:44.724561 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:44.724752 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:44.724762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.725883 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.725890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:44.726971 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:44.726985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.727866 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.727873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:44.728072 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:44.728735 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:44.728754 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:44.732440 (+ 3686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":380,"threads_started":1}
W20260808 15:49:44.732947 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:44.722830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:53589 (local address 127.27.84.254:44445)
0808 15:49:44.723041 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:44.723046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:44.723588 (+ 542us) server_negotiation.cc:407] Connection header received
0808 15:49:44.723807 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:44.723812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:44.723901 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:44.724035 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:44.724920 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.725702 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.727138 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:44.727646 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:44.727802 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:44.732655 (+ 4853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:53589: BlockingRecv error: recv got EOF from 127.27.84.194:53589 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 08 15:49:45 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:45 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:45.091430 10092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:45.081762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:57197)
0808 15:49:45.082284 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:45.082301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:45.082429 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:45.082818 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:45.082822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:45.083241 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:45.083400 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:45.083409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.084880 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.084886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:45.086095 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:45.086109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.086860 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.086865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:45.086999 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:45.087616 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:45.087633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:45.091193 (+ 3560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":269,"threads_started":1}
W20260808 15:49:45.091653 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:45.081874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:57197 (local address 127.27.84.254:44445)
0808 15:49:45.082149 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:45.082157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:45.082393 (+ 236us) server_negotiation.cc:407] Connection header received
0808 15:49:45.082539 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:45.082544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:45.082642 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:45.082741 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:45.083575 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.084692 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.086241 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.086727 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.086839 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:45.091407 (+ 4568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:57197: BlockingRecv error: recv got EOF from 127.27.84.193:57197 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 08 15:49:45 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:45 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:45.398802 10094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:45.388092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:57813)
0808 15:49:45.388779 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:45.388798 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:45.388986 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:45.389378 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:45.389383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:45.389881 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:45.390082 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:45.390092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.391339 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.391347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:45.392406 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:45.392419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.393284 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.393292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:45.393500 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:45.394229 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:45.394250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:45.398386 (+ 4136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":396,"threads_started":1}
W20260808 15:49:45.398916 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:45.388131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:57813 (local address 127.27.84.254:44445)
0808 15:49:45.388437 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:45.388446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:45.388910 (+ 464us) server_negotiation.cc:407] Connection header received
0808 15:49:45.389130 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:45.389136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:45.389216 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:45.389347 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:45.390249 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.391176 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.392568 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.393066 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.393219 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:45.398640 (+ 5421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:57813: BlockingRecv error: recv got EOF from 127.27.84.195:57813 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 08 15:49:45 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:45 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:45.745203 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:45.734029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:49025)
0808 15:49:45.734908 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:45.734926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:45.735052 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:45.735526 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:45.735531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:45.736088 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:45.736332 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:45.736351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.737640 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.737644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:45.738579 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:45.738593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.739563 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.739568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:45.739723 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:45.740450 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:45.740471 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:45.744884 (+ 4413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":346,"threads_started":1}
W20260808 15:49:45.745740 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:45.734324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:49025 (local address 127.27.84.254:44445)
0808 15:49:45.734576 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:45.734583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:45.735037 (+ 454us) server_negotiation.cc:407] Connection header received
0808 15:49:45.735218 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:45.735223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:45.735288 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:45.735388 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:45.736493 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.737465 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.738819 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:45.739379 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:45.739505 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:45.745422 (+ 5917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:49025: BlockingRecv error: recv got EOF from 127.27.84.194:49025 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 08 15:49:46 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:46 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.193@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:46.103427 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:46.092879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.193:60563)
0808 15:49:46.093510 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:46.093532 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:46.093702 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:46.094126 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:46.094131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:46.094765 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:46.095010 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:46.095023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.096322 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.096326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:46.097441 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:46.097455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.098294 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.098298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:46.098443 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:46.099084 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:46.099101 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:46.103167 (+ 4066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":324,"threads_started":1}
W20260808 15:49:46.103690 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:46.093020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:60563 (local address 127.27.84.254:44445)
0808 15:49:46.093304 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:46.093312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:46.093647 (+ 335us) server_negotiation.cc:407] Connection header received
0808 15:49:46.093870 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:46.093879 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:46.093963 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:46.094057 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:46.095194 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.096174 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.097614 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.098100 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.098209 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:46.103420 (+ 5211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.193:60563: BlockingRecv error: recv got EOF from 127.27.84.193:60563 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 08 15:49:46 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:46 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.195@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:46.409947 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:46.399932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.195:38163)
0808 15:49:46.400691 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:46.400724 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:46.400886 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:46.401339 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:46.401343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:46.401924 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:46.402088 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:46.402097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.403285 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.403289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:46.404221 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:46.404234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.405058 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.405063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:46.405189 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:46.405805 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:46.405824 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:46.409691 (+ 3867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":491,"threads_started":1}
W20260808 15:49:46.410337 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:46.400023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:38163 (local address 127.27.84.254:44445)
0808 15:49:46.400445 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:46.400454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:46.400810 (+ 356us) server_negotiation.cc:407] Connection header received
0808 15:49:46.401060 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:46.401065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:46.401155 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:46.401274 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:46.402261 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.403116 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.404393 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.404910 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.405032 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:46.410035 (+ 5003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.195:38163: BlockingRecv error: recv got EOF from 127.27.84.195:38163 (error 108)
Metrics: {"server-negotiator.queue_time_us":249}
Aug 08 15:49:46 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
Aug 08 15:49:46 dist-test-slave-twwt krb5kdc[8421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.84.194@KRBTEST.COM for oryx/127.27.84.254@KRBTEST.COM, Server not found in Kerberos database
I20260808 15:49:46.757383 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:46.746778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44445 (local address 127.27.84.194:50617)
0808 15:49:46.747440 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:46.747459 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:46.747582 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:46.748003 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:46.748007 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:46.748434 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:46.748641 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:46.748650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.749817 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.749822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:46.750990 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:46.751006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.751766 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.751771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:46.751924 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:46.752665 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:46.752685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:46.757073 (+ 4388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.84.254:44445: Server oryx/127.27.84.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":325,"threads_started":1}
W20260808 15:49:46.757607 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:46.746888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:50617 (local address 127.27.84.254:44445)
0808 15:49:46.747104 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:46.747108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:46.747538 (+ 430us) server_negotiation.cc:407] Connection header received
0808 15:49:46.747688 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:46.747691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:46.747759 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:46.747894 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:46.748797 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.749653 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.751155 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:46.751608 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:46.751718 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:46.757346 (+ 5628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.84.194:50617: BlockingRecv error: recv got EOF from 127.27.84.194:50617 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:49:47.088016 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9344
I20260808 15:49:47.101948 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:47.089326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.27.84.193:41283)
0808 15:49:47.089955 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:47.089963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:47.089991 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:49:47.090070 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:47.090077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:47.090330 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:47.090499 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:47.092018 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.093088 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.094335 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.095183 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.095321 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:47.098076 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:47.098086 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:47.098101 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:47.098147 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:47.101359 (+ 3212us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.84.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:49:47.101674 (+ 315us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33202: Request ticket server kudu/127.27.84.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":290,"threads_started":1}
W20260808 15:49:47.102082 10107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33202: Request ticket server kudu/127.27.84.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:49:47.102144 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9344: Not authorized: Client connection negotiation failed: client connection to 127.27.84.193:41283: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.84.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:49:47.102550 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 9344
I20260808 15:49:47.134673 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9582
I20260808 15:49:47.148895 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:47.136100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.27.84.194:38611)
0808 15:49:47.136795 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:47.136802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:47.136834 (+ 32us) server_negotiation.cc:407] Connection header received
0808 15:49:47.136898 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:47.136905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:47.137143 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:47.137428 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:47.138959 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.140079 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.141436 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.142036 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.142192 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:47.145144 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:47.145154 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:47.145170 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:47.145219 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:47.148079 (+ 2860us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.84.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:49:47.148495 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52944: Request ticket server kudu/127.27.84.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":362,"threads_started":1}
W20260808 15:49:47.148931 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9582: Not authorized: Client connection negotiation failed: client connection to 127.27.84.194:38611: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.84.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:49:47.149021 10109 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52944: Request ticket server kudu/127.27.84.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:49:47.149088 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 9582
I20260808 15:49:47.179541 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9799
I20260808 15:49:47.193903 10110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:47.180821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52586 (local address 127.27.84.195:34891)
0808 15:49:47.181527 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:47.181535 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:47.181563 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:49:47.181639 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:47.181646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:47.181937 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:47.182174 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:47.183745 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.184818 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.186158 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.186848 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.186999 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:47.189715 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:47.189724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:47.189741 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:47.189790 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:47.193008 (+ 3218us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.84.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0808 15:49:47.193466 (+ 458us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52586: Request ticket server kudu/127.27.84.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":364,"threads_started":1}
W20260808 15:49:47.193912 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9799: Not authorized: Client connection negotiation failed: client connection to 127.27.84.195:34891: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.84.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260808 15:49:47.194044 10110 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52586: Request ticket server kudu/127.27.84.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260808 15:49:47.194105 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 9799
I20260808 15:49:47.223779 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 9240
I20260808 15:49:47.225590 9335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:47.346869 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 9240
2026-08-08T15:49:47Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22385 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-08T15:49:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:47Z Disabled control of system clock
I20260808 15:49:47.450234 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:39765
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260808 15:49:47.808051 10117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:47.808379 10117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:47.808467 10117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:47.815759 10117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:47.815886 10117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:47.815925 10117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:47.815986 10117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:47.816031 10117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:47.825443 10117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:39765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:47.826835 10117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:47.828706 10117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:47.841110 10126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:47.841146 10127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:47.841184 10129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:47.842010 10117 server_base.cc:1034] running on GCE node
I20260808 15:49:47.842928 10117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:47.844367 10117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:47.845587 10117 hybrid_clock.cc:648] HybridClock initialized: now 1786204187845542 us; error 27 us; skew 500 ppm
I20260808 15:49:47.845924 10117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:47.848248 10117 webserver.cc:460] Webserver started at http://127.27.84.254:37343/ using document root <none> and password file <none>
I20260808 15:49:47.848954 10117 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:47.849109 10117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:47.849429 10117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:47.852490 10117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "194c9ddb2a714ec89fafece05bcbc2ed"
format_stamp: "Formatted at 2026-08-08 15:49:47 on dist-test-slave-twwt"
server_key: "8e63ee8d917024e94a8faa862e2640da"
server_key_iv: "dd7cbcebcfc5c985c101e88ce458c77b"
server_key_version: "encryptionkey@0"
I20260808 15:49:47.853682 10117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "194c9ddb2a714ec89fafece05bcbc2ed"
format_stamp: "Formatted at 2026-08-08 15:49:47 on dist-test-slave-twwt"
server_key: "8e63ee8d917024e94a8faa862e2640da"
server_key_iv: "dd7cbcebcfc5c985c101e88ce458c77b"
server_key_version: "encryptionkey@0"
I20260808 15:49:47.859668 10117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 15:49:47.863408 10137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:47.864938 10117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:49:47.865103 10117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "194c9ddb2a714ec89fafece05bcbc2ed"
format_stamp: "Formatted at 2026-08-08 15:49:47 on dist-test-slave-twwt"
server_key: "8e63ee8d917024e94a8faa862e2640da"
server_key_iv: "dd7cbcebcfc5c985c101e88ce458c77b"
server_key_version: "encryptionkey@0"
I20260808 15:49:47.865324 10117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:47.877774 10117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:47.878713 10117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:47.879005 10117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:47.899140 10117 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:39765
I20260808 15:49:47.899158 10216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:39765 every 8 connection(s)
I20260808 15:49:47.900658 10117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:47.905061 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:47.910841 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 10117
I20260808 15:49:47.911049 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:47.911895 27987 external_mini_cluster.cc:1428] Setting key a449c4a7bb5a0ec360a580ac040c6af0
I20260808 15:49:47.917487 10218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed: Bootstrap starting.
I20260808 15:49:47.921144 10218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:47.922448 10218 log.cc:824] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:47.925988 10218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed: No bootstrap required, opened a new log
I20260808 15:49:47.927573 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:47.914649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.27.84.254:39765)
0808 15:49:47.915888 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:47.915900 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:47.915949 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:49:47.917626 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:47.917663 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:47.918132 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:47.918460 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:47.919255 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.920595 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.922660 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:47.923749 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:47.923963 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:47.925160 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:47.925169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:47.925183 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:47.925717 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:47.925801 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:47.925823 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:47.925909 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:47.926212 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 15:49:47.926565 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":453,"threads_started":1}
I20260808 15:49:47.933826 10218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39765 } }
I20260808 15:49:47.934207 10218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:47.934341 10218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194c9ddb2a714ec89fafece05bcbc2ed, State: Initialized, Role: FOLLOWER
I20260808 15:49:47.934954 10218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39765 } }
I20260808 15:49:47.935187 10218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:47.935346 10218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:47.935556 10218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:47.937819 10218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39765 } }
I20260808 15:49:47.938354 10218 leader_election.cc:302] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 194c9ddb2a714ec89fafece05bcbc2ed; no voters:
I20260808 15:49:47.939033 10218 leader_election.cc:288] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:47.939254 10228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:47.940145 10228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [term 1 LEADER]: Becoming Leader. State: Replica: 194c9ddb2a714ec89fafece05bcbc2ed, State: Running, Role: LEADER
I20260808 15:49:47.940697 10228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39765 } }
I20260808 15:49:47.941326 10218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:47.944145 10229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39765 } } }
I20260808 15:49:47.944629 10229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:47.945182 10230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 194c9ddb2a714ec89fafece05bcbc2ed. Latest consensus state: current_term: 1 leader_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "194c9ddb2a714ec89fafece05bcbc2ed" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39765 } } }
I20260808 15:49:47.945554 10230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:47.945861 10239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:47.964071 10239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:47.977671 10239 catalog_manager.cc:1347] Generated new cluster ID: bee27efdbc83493e99000f74fd3a0a54
I20260808 15:49:47.977913 10239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:47.991997 10239 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:47.993749 10239 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:48.002979 10239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 194c9ddb2a714ec89fafece05bcbc2ed: Generated new TSK 0
I20260808 15:49:48.003907 10239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:49:48.008555 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39765
--builtin_ntp_servers=127.27.84.212:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260808 15:49:48.394439 10256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:48.394855 10256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:48.394956 10256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:48.402343 10256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:48.402467 10256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:48.402578 10256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:48.412231 10256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:48.413668 10256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:48.415686 10256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:48.429739 10267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:48.429842 10266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:48.429850 10269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:48.431018 10256 server_base.cc:1034] running on GCE node
I20260808 15:49:48.431675 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:48.432718 10256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:48.433946 10256 hybrid_clock.cc:648] HybridClock initialized: now 1786204188433872 us; error 63 us; skew 500 ppm
I20260808 15:49:48.434312 10256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:48.440717 10256 webserver.cc:460] Webserver started at http://127.27.84.193:46815/ using document root <none> and password file <none>
I20260808 15:49:48.441670 10256 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:48.441810 10256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:48.442236 10256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:48.445770 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "19f2614129464ac8a0008652e354335e"
format_stamp: "Formatted at 2026-08-08 15:49:48 on dist-test-slave-twwt"
server_key: "02010f89d8ad959f80de7e68d7ccbf76"
server_key_iv: "4fa01d69c79c1a5aa0cca116ce635dc8"
server_key_version: "encryptionkey@0"
I20260808 15:49:48.446911 10256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "19f2614129464ac8a0008652e354335e"
format_stamp: "Formatted at 2026-08-08 15:49:48 on dist-test-slave-twwt"
server_key: "02010f89d8ad959f80de7e68d7ccbf76"
server_key_iv: "4fa01d69c79c1a5aa0cca116ce635dc8"
server_key_version: "encryptionkey@0"
I20260808 15:49:48.452737 10256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:49:48.456723 10278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:48.458202 10256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:48.458401 10256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "19f2614129464ac8a0008652e354335e"
format_stamp: "Formatted at 2026-08-08 15:49:48 on dist-test-slave-twwt"
server_key: "02010f89d8ad959f80de7e68d7ccbf76"
server_key_iv: "4fa01d69c79c1a5aa0cca116ce635dc8"
server_key_version: "encryptionkey@0"
I20260808 15:49:48.458684 10256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:48.475937 10256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:48.477000 10256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:48.477366 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:48.478451 10256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:48.480103 10256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:48.480201 10256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:48.480377 10256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:48.480468 10256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:48.517554 10256 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:34717
I20260808 15:49:48.517612 10448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:34717 every 8 connection(s)
I20260808 15:49:48.519066 10256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:48.527688 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 10256
I20260808 15:49:48.527920 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:48.528828 27987 external_mini_cluster.cc:1428] Setting key 282b25a3f287bfb5aaf45442fde6955c
I20260808 15:49:48.532071 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39765
--builtin_ntp_servers=127.27.84.212:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 15:49:48.533114 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:48.522098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:58271 (local address 127.27.84.254:39765)
0808 15:49:48.522678 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:48.522683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:48.523933 (+ 1250us) server_negotiation.cc:407] Connection header received
0808 15:49:48.525795 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:48.525801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:48.525914 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:48.526042 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:48.527268 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:48.528074 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:48.529908 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:48.530337 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:48.530479 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:48.532076 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:48.532081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:48.532084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:48.532102 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:48.532174 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:48.532194 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:48.532301 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:48.532595 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 15:49:48.532710 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":301,"threads_started":1}
I20260808 15:49:48.534186 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:48.522684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39765 (local address 127.27.84.193:58271)
0808 15:49:48.523710 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:48.523796 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:48.525497 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:48.526296 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:48.526306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:48.526398 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:48.527077 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:48.527095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:48.528224 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:48.528232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:48.529742 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:48.529756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:48.530574 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:48.530581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:48.530890 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:48.531660 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:49:48.531666 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:48.531736 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:49:48.531850 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:48.532361 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:48.532371 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:48.532378 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:49:48.532565 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 15:49:48.532997 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":332,"threads_started":1}
I20260808 15:49:48.535802 10449 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39765
I20260808 15:49:48.536152 10449 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:48.537178 10449 heartbeater.cc:511] Master 127.27.84.254:39765 requested a full tablet report, sending...
I20260808 15:49:48.539620 10165 ts_manager.cc:194] Registered new tserver with Master: 19f2614129464ac8a0008652e354335e (127.27.84.193:34717)
I20260808 15:49:48.541357 10165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.193:58271
W20260808 15:49:48.890936 10453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:48.891233 10453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:48.891391 10453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:48.899536 10453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:48.899662 10453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:48.899837 10453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:48.915066 10453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:48.916750 10453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:48.919196 10453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:48.934197 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:48.934290 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:48.934201 10466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:48.935113 10453 server_base.cc:1034] running on GCE node
I20260808 15:49:48.935889 10453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:48.937007 10453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:48.938328 10453 hybrid_clock.cc:648] HybridClock initialized: now 1786204188938246 us; error 70 us; skew 500 ppm
I20260808 15:49:48.938781 10453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:48.941404 10453 webserver.cc:460] Webserver started at http://127.27.84.194:42415/ using document root <none> and password file <none>
I20260808 15:49:48.942266 10453 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:48.942412 10453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:48.942855 10453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:48.946168 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "2b20313d475e4074813a438016c3c0f4"
format_stamp: "Formatted at 2026-08-08 15:49:48 on dist-test-slave-twwt"
server_key: "9bb713854458e5e869650e8d4a83ae05"
server_key_iv: "b3c222bc183de4174392e0e208323743"
server_key_version: "encryptionkey@0"
I20260808 15:49:48.947095 10453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "2b20313d475e4074813a438016c3c0f4"
format_stamp: "Formatted at 2026-08-08 15:49:48 on dist-test-slave-twwt"
server_key: "9bb713854458e5e869650e8d4a83ae05"
server_key_iv: "b3c222bc183de4174392e0e208323743"
server_key_version: "encryptionkey@0"
I20260808 15:49:48.952782 10453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:48.956760 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:48.958226 10453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:49:48.958428 10453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "2b20313d475e4074813a438016c3c0f4"
format_stamp: "Formatted at 2026-08-08 15:49:48 on dist-test-slave-twwt"
server_key: "9bb713854458e5e869650e8d4a83ae05"
server_key_iv: "b3c222bc183de4174392e0e208323743"
server_key_version: "encryptionkey@0"
I20260808 15:49:48.958694 10453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:48.989121 10453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:48.990173 10453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:48.990499 10453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:48.991564 10453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:48.993147 10453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:48.993283 10453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:48.993436 10453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:48.993541 10453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:49.029474 10453 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:34443
I20260808 15:49:49.029569 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:34443 every 8 connection(s)
I20260808 15:49:49.030934 10453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:49.036629 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 10453
I20260808 15:49:49.036805 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:49.037427 27987 external_mini_cluster.cc:1428] Setting key b19d39af6e72cfc2434f24a760a9842f
I20260808 15:49:49.040815 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39765
--builtin_ntp_servers=127.27.84.212:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260808 15:49:49.045815 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.034671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:57061 (local address 127.27.84.254:39765)
0808 15:49:49.035154 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:49.035160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:49.035541 (+ 381us) server_negotiation.cc:407] Connection header received
0808 15:49:49.037503 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:49.037509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:49.037599 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:49.037721 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:49.039140 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.040224 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.042047 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.042874 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.043038 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.044571 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:49.044580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:49.044587 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:49.044614 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:49.044670 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:49.044690 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:49.044790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:49.045217 (+ 427us) server_negotiation.cc:299] Negotiation successful
0808 15:49:49.045359 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":947,"threads_started":1}
I20260808 15:49:49.046970 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.034214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39765 (local address 127.27.84.194:57061)
0808 15:49:49.035355 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:49.035415 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:49.037227 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:49.037949 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:49.037958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:49.038082 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:49.038943 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:49.038970 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.040391 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.040395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:49.041904 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:49.041918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.043143 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.043149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:49.043381 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.044170 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:49:49.044173 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:49.044217 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:49:49.044364 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:49.044897 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:49.044907 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:49.044915 (+ 8us) client_negotiation.cc:769] Sending connection context
0808 15:49:49.045183 (+ 268us) client_negotiation.cc:240] Negotiation successful
0808 15:49:49.045634 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":520,"threads_started":1}
I20260808 15:49:49.048686 10659 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39765
I20260808 15:49:49.049047 10659 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:49.050083 10659 heartbeater.cc:511] Master 127.27.84.254:39765 requested a full tablet report, sending...
I20260808 15:49:49.051868 10165 ts_manager.cc:194] Registered new tserver with Master: 2b20313d475e4074813a438016c3c0f4 (127.27.84.194:34443)
I20260808 15:49:49.052809 10165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.194:57061
W20260808 15:49:49.402364 10664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:49.402727 10664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:49.402858 10664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:49.410300 10664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:49.410419 10664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:49.410526 10664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:49.420014 10664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:39765
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:49.421303 10664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:49.423318 10664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:49.436996 10675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:49.437016 10680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:49.437650 10664 server_base.cc:1034] running on GCE node
W20260808 15:49:49.437788 10671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:49.438664 10664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:49.439653 10664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:49.440896 10664 hybrid_clock.cc:648] HybridClock initialized: now 1786204189440831 us; error 59 us; skew 500 ppm
I20260808 15:49:49.441233 10664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:49.443946 10664 webserver.cc:460] Webserver started at http://127.27.84.195:42673/ using document root <none> and password file <none>
I20260808 15:49:49.444741 10664 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:49.444881 10664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:49.445225 10664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:49.448217 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "96ee69e8cc2748549a1886a3f0d7b419"
format_stamp: "Formatted at 2026-08-08 15:49:49 on dist-test-slave-twwt"
server_key: "787d08a4e843b0c23b84299666e41796"
server_key_iv: "65723138389ce941005ceec1b28857d1"
server_key_version: "encryptionkey@0"
I20260808 15:49:49.449019 10664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "96ee69e8cc2748549a1886a3f0d7b419"
format_stamp: "Formatted at 2026-08-08 15:49:49 on dist-test-slave-twwt"
server_key: "787d08a4e843b0c23b84299666e41796"
server_key_iv: "65723138389ce941005ceec1b28857d1"
server_key_version: "encryptionkey@0"
I20260808 15:49:49.454315 10664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:49:49.458047 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:49.459453 10664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:49.459620 10664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "96ee69e8cc2748549a1886a3f0d7b419"
format_stamp: "Formatted at 2026-08-08 15:49:49 on dist-test-slave-twwt"
server_key: "787d08a4e843b0c23b84299666e41796"
server_key_iv: "65723138389ce941005ceec1b28857d1"
server_key_version: "encryptionkey@0"
I20260808 15:49:49.459848 10664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:49.474761 10664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:49.475921 10664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:49.476279 10664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:49.477331 10664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:49.479558 10664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:49.479684 10664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:49.479851 10664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:49.479959 10664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:49.517550 10664 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:39631
I20260808 15:49:49.517653 10888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:39631 every 8 connection(s)
I20260808 15:49:49.519074 10664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:49.524479 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 10664
I20260808 15:49:49.524626 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:49.525143 27987 external_mini_cluster.cc:1428] Setting key 5257228ec2699ae811ae03bc4cce3dbc
I20260808 15:49:49.532982 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.522202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:56083 (local address 127.27.84.254:39765)
0808 15:49:49.522460 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:49.522467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:49.523927 (+ 1460us) server_negotiation.cc:407] Connection header received
0808 15:49:49.526019 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:49.526025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:49.526143 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:49.526283 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:49.527715 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.528675 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.530317 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.530854 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.530963 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.532294 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:49.532300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:49.532303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:49.532321 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:49.532354 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:49.532366 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:49.532442 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:49.532658 (+ 216us) server_negotiation.cc:299] Negotiation successful
0808 15:49:49.532751 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 15:49:49.534070 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.522802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:39765 (local address 127.27.84.195:56083)
0808 15:49:49.523747 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:49.523789 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:49.525641 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:49.526596 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:49.526608 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:49.526764 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:49.527511 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:49.527543 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.528838 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.528844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:49.530147 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:49.530160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.531064 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.531069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:49.531268 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.531977 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:49:49.531980 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:49.532019 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:49:49.532096 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:49.532464 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:49.532473 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:49.532479 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 15:49:49.532653 (+ 174us) client_negotiation.cc:240] Negotiation successful
0808 15:49:49.533040 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":327,"threads_started":1}
I20260808 15:49:49.535446 10889 heartbeater.cc:347] Connected to a master server at 127.27.84.254:39765
I20260808 15:49:49.535732 10889 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:49.536474 10889 heartbeater.cc:511] Master 127.27.84.254:39765 requested a full tablet report, sending...
I20260808 15:49:49.537904 10165 ts_manager.cc:194] Registered new tserver with Master: 96ee69e8cc2748549a1886a3f0d7b419 (127.27.84.195:39631)
I20260808 15:49:49.538806 10165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.195:56083
I20260808 15:49:49.541419 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:49:49.545432 10449 heartbeater.cc:503] Master 127.27.84.254:39765 was elected leader, sending a full tablet report...
W20260808 15:49:49.549587 10662 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:49.547804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.27.84.254:39765)
0808 15:49:49.548055 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:49.548064 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:49.548535 (+ 471us) server_negotiation.cc:407] Connection header received
0808 15:49:49.548678 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:49.548682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:49.548744 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:49.548863 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:49.549319 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60588: BlockingRecv error: recv got EOF from 127.0.0.1:60588 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:49:49.550423 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 10256
I20260808 15:49:49.558876 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.551448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.27.84.193:34717)
0808 15:49:49.552062 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:49.552069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:49.552094 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:49:49.552147 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:49.552153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:49.552378 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:49.552550 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:49.553010 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.554043 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.555397 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.556003 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.556152 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.557362 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:49.557371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:49.557385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:49.557979 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:49.558028 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:49.558049 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:49.558181 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:49.558356 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 15:49:49.558490 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":250,"threads_started":1}
I20260808 15:49:49.560304 10444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:49.643492 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 10256
I20260808 15:49:49.669129 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 10453
I20260808 15:49:49.677084 10913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.670213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53590 (local address 127.27.84.194:34443)
0808 15:49:49.670895 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:49.670902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:49.670933 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:49:49.670994 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:49.671001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:49.671224 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:49.671362 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:49.671770 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.672793 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.673982 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.674548 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.674711 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.675736 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:49.675745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:49.675758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:49.676356 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:49.676403 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:49.676422 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:49.676527 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:49.676668 (+ 141us) server_negotiation.cc:299] Negotiation successful
0808 15:49:49.676779 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":370,"threads_started":1}
I20260808 15:49:49.678375 10647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:49.761835 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 10453
I20260808 15:49:49.787777 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 10664
I20260808 15:49:49.796526 10916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:49.788940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.27.84.195:39631)
0808 15:49:49.789654 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:49.789661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:49.789692 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:49:49.789760 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:49.789766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:49.790024 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:49.790154 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:49.790588 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.791839 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.793147 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:49.793704 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:49.793811 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:49.795038 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:49.795045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:49.795059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:49.795648 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:49.795700 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:49.795718 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:49.795838 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:49.795975 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 15:49:49.796127 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":356,"threads_started":1}
I20260808 15:49:49.797840 10880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:49.882597 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 10664
I20260808 15:49:49.908327 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 10117
I20260808 15:49:49.909569 10212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:50.020256 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 10117
2026-08-08T15:49:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2671 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-08T15:49:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:50Z Disabled control of system clock
I20260808 15:49:50.122995 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44955
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:44955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 15:49:50.481225 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:50.481612 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:50.481773 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:50.489179 10923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:50.489348 10923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:50.489449 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:50.489562 10923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:50.489650 10923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:50.499361 10923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:44955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:44955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:50.500878 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:50.502867 10923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:50.515187 10938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:50.515233 10935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:50.515233 10942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:50.516285 10923 server_base.cc:1034] running on GCE node
I20260808 15:49:50.517119 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:50.518599 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:50.519889 10923 hybrid_clock.cc:648] HybridClock initialized: now 1786204190519833 us; error 59 us; skew 500 ppm
I20260808 15:49:50.520299 10923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:50.522758 10923 webserver.cc:460] Webserver started at http://127.27.84.254:35729/ using document root <none> and password file <none>
I20260808 15:49:50.523491 10923 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:50.523627 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:50.523970 10923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:50.526999 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "6c053de4dc1947aaa9e1f1f4327d83d5"
format_stamp: "Formatted at 2026-08-08 15:49:50 on dist-test-slave-twwt"
server_key: "1a98f2b5e05dd8f51edff87ae01c2858"
server_key_iv: "e1b188f5b73e36d564371613048ec96a"
server_key_version: "encryptionkey@0"
I20260808 15:49:50.527829 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "6c053de4dc1947aaa9e1f1f4327d83d5"
format_stamp: "Formatted at 2026-08-08 15:49:50 on dist-test-slave-twwt"
server_key: "1a98f2b5e05dd8f51edff87ae01c2858"
server_key_iv: "e1b188f5b73e36d564371613048ec96a"
server_key_version: "encryptionkey@0"
I20260808 15:49:50.533039 10923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 15:49:50.536509 10952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:50.537830 10923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:49:50.538017 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "6c053de4dc1947aaa9e1f1f4327d83d5"
format_stamp: "Formatted at 2026-08-08 15:49:50 on dist-test-slave-twwt"
server_key: "1a98f2b5e05dd8f51edff87ae01c2858"
server_key_iv: "e1b188f5b73e36d564371613048ec96a"
server_key_version: "encryptionkey@0"
I20260808 15:49:50.538239 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:50.544119 10923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:50.544901 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:50.545215 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:50.564105 10923 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:44955
I20260808 15:49:50.564141 11047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:44955 every 8 connection(s)
I20260808 15:49:50.565567 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:50.569511 11048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:50.572052 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 10923
I20260808 15:49:50.572289 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:50.573127 27987 external_mini_cluster.cc:1428] Setting key 30b2d89fca77f2df34f5d250ca360272
I20260808 15:49:50.582304 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:50.575498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.27.84.254:44955)
0808 15:49:50.576689 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:50.576700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:50.576742 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:49:50.578444 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:50.578467 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:50.578983 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:50.579424 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:50.580603 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:50.580609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:50.580630 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:50.580644 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:50.580702 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:50.580718 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:50.580981 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:50.581118 (+ 137us) server_negotiation.cc:299] Negotiation successful
0808 15:49:50.581374 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":492,"threads_started":1}
I20260808 15:49:50.582901 11048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5: Bootstrap starting.
I20260808 15:49:50.586413 11048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:50.587515 11048 log.cc:824] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:50.590579 11048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5: No bootstrap required, opened a new log
I20260808 15:49:50.598579 11048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44955 } }
I20260808 15:49:50.599124 11048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:50.599206 11048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c053de4dc1947aaa9e1f1f4327d83d5, State: Initialized, Role: FOLLOWER
I20260808 15:49:50.599736 11048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44955 } }
I20260808 15:49:50.599951 11048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:50.600055 11048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:50.600171 11048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:50.602353 11048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44955 } }
I20260808 15:49:50.602963 11048 leader_election.cc:302] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c053de4dc1947aaa9e1f1f4327d83d5; no voters:
I20260808 15:49:50.603590 11048 leader_election.cc:288] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:50.603703 11055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:50.605162 11055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [term 1 LEADER]: Becoming Leader. State: Replica: 6c053de4dc1947aaa9e1f1f4327d83d5, State: Running, Role: LEADER
I20260808 15:49:50.605738 11055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44955 } }
I20260808 15:49:50.606230 11048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:50.608986 11056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44955 } } }
I20260808 15:49:50.608979 11057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c053de4dc1947aaa9e1f1f4327d83d5. Latest consensus state: current_term: 1 leader_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c053de4dc1947aaa9e1f1f4327d83d5" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 44955 } } }
I20260808 15:49:50.609855 11056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:50.609872 11057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:50.611802 11066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:50.617542 11066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:50.626667 11066 catalog_manager.cc:1347] Generated new cluster ID: 10ed674b7bac46d2add50b428539985e
I20260808 15:49:50.626791 11066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:50.653033 11066 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:50.653554 11066 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:50.663182 11066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c053de4dc1947aaa9e1f1f4327d83d5: Generated new TSK 0
I20260808 15:49:50.664068 11066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:49:50.669984 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44955
--builtin_ntp_servers=127.27.84.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260808 15:49:51.053642 11088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:51.053953 11088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:51.054105 11088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:51.061502 11088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:51.061609 11088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:51.061801 11088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:51.071193 11088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:51.072614 11088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:51.074494 11088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:51.088467 11094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:51.088490 11099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:51.088786 11102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:51.089802 11088 server_base.cc:1034] running on GCE node
I20260808 15:49:51.090425 11088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:51.091475 11088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:51.092718 11088 hybrid_clock.cc:648] HybridClock initialized: now 1786204191092642 us; error 69 us; skew 500 ppm
I20260808 15:49:51.093050 11088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:51.095507 11088 webserver.cc:460] Webserver started at http://127.27.84.193:32983/ using document root <none> and password file <none>
I20260808 15:49:51.096282 11088 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:51.096416 11088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:51.096740 11088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:51.099798 11088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "8682dcf3452b4a22b325b824ba3c4533"
format_stamp: "Formatted at 2026-08-08 15:49:51 on dist-test-slave-twwt"
server_key: "895804cee352a223244dbc89bfa96a82"
server_key_iv: "bf660a0090d9992f02ba23ac6e07d19e"
server_key_version: "encryptionkey@0"
I20260808 15:49:51.100663 11088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "8682dcf3452b4a22b325b824ba3c4533"
format_stamp: "Formatted at 2026-08-08 15:49:51 on dist-test-slave-twwt"
server_key: "895804cee352a223244dbc89bfa96a82"
server_key_iv: "bf660a0090d9992f02ba23ac6e07d19e"
server_key_version: "encryptionkey@0"
I20260808 15:49:51.106241 11088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:49:51.110036 11113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:51.111999 11088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:49:51.112188 11088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "8682dcf3452b4a22b325b824ba3c4533"
format_stamp: "Formatted at 2026-08-08 15:49:51 on dist-test-slave-twwt"
server_key: "895804cee352a223244dbc89bfa96a82"
server_key_iv: "bf660a0090d9992f02ba23ac6e07d19e"
server_key_version: "encryptionkey@0"
I20260808 15:49:51.112432 11088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:51.118729 11088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:51.119573 11088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:51.119871 11088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:51.120877 11088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:51.122351 11088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:51.122480 11088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:51.122704 11088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:51.122807 11088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260808 15:49:51.159157 11088 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:35443
I20260808 15:49:51.159210 11286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:35443 every 8 connection(s)
I20260808 15:49:51.160677 11088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:51.167694 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 11088
I20260808 15:49:51.167958 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:51.168869 27987 external_mini_cluster.cc:1428] Setting key a3722ee4c97888090e6796a3958340a8
I20260808 15:49:51.170843 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:51.163694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:45129 (local address 127.27.84.254:44955)
0808 15:49:51.164140 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:51.164144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:51.165772 (+ 1628us) server_negotiation.cc:407] Connection header received
0808 15:49:51.168062 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:51.168069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:51.168149 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:51.168265 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:51.169974 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:51.169978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:51.169982 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:51.170001 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:51.170047 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:51.170060 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:51.170217 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:51.170418 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 15:49:51.170495 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260808 15:49:51.172251 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:51.164368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44955 (local address 127.27.84.193:45129)
0808 15:49:51.165581 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:51.165623 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:51.167603 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:51.168594 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:51.168608 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:51.168779 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:51.169616 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:49:51.169619 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:51.169665 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:49:51.169772 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:51.170169 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:51.170182 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:51.170193 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:49:51.170516 (+ 323us) client_negotiation.cc:240] Negotiation successful
0808 15:49:51.170872 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":347,"threads_started":1}
I20260808 15:49:51.172309 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44955
--builtin_ntp_servers=127.27.84.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 15:49:51.174029 11287 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44955
I20260808 15:49:51.174394 11287 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:51.175125 11287 heartbeater.cc:511] Master 127.27.84.254:44955 requested a full tablet report, sending...
I20260808 15:49:51.177448 10984 ts_manager.cc:194] Registered new tserver with Master: 8682dcf3452b4a22b325b824ba3c4533 (127.27.84.193:35443)
W20260808 15:49:51.527518 11294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:51.527812 11294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:51.527985 11294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:51.535481 11294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:51.535588 11294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:51.535759 11294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:51.545357 11294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:51.546739 11294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:51.548650 11294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:51.562994 11303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:51.563107 11302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:51.563131 11305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:51.567821 11294 server_base.cc:1034] running on GCE node
I20260808 15:49:51.568773 11294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:51.569896 11294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:51.571141 11294 hybrid_clock.cc:648] HybridClock initialized: now 1786204191571074 us; error 59 us; skew 500 ppm
I20260808 15:49:51.571506 11294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:51.574191 11294 webserver.cc:460] Webserver started at http://127.27.84.194:36557/ using document root <none> and password file <none>
I20260808 15:49:51.575080 11294 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:51.575224 11294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:51.575570 11294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:51.578586 11294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "e8fdf019672740eca550838e817691bd"
format_stamp: "Formatted at 2026-08-08 15:49:51 on dist-test-slave-twwt"
server_key: "5ee15e228391f2f9bf149dd95b2606ad"
server_key_iv: "2a43ef29d884cc5c15f4707edf5652c2"
server_key_version: "encryptionkey@0"
I20260808 15:49:51.579478 11294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "e8fdf019672740eca550838e817691bd"
format_stamp: "Formatted at 2026-08-08 15:49:51 on dist-test-slave-twwt"
server_key: "5ee15e228391f2f9bf149dd95b2606ad"
server_key_iv: "2a43ef29d884cc5c15f4707edf5652c2"
server_key_version: "encryptionkey@0"
I20260808 15:49:51.585054 11294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:49:51.589157 11312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:51.590792 11294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:49:51.590971 11294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "e8fdf019672740eca550838e817691bd"
format_stamp: "Formatted at 2026-08-08 15:49:51 on dist-test-slave-twwt"
server_key: "5ee15e228391f2f9bf149dd95b2606ad"
server_key_iv: "2a43ef29d884cc5c15f4707edf5652c2"
server_key_version: "encryptionkey@0"
I20260808 15:49:51.591190 11294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:51.597263 11294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:51.598069 11294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:51.598351 11294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:51.599411 11294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:51.600840 11294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:51.600953 11294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:51.601092 11294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:51.601174 11294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:51.639183 11294 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:36801
I20260808 15:49:51.639281 11482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:36801 every 8 connection(s)
I20260808 15:49:51.640617 11294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:51.645414 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 11294
I20260808 15:49:51.645640 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:51.646263 27987 external_mini_cluster.cc:1428] Setting key 74cb7408a9bbd8d3953eb7f3710c2c87
I20260808 15:49:51.649788 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44955
--builtin_ntp_servers=127.27.84.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260808 15:49:51.650851 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:51.643787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:43601 (local address 127.27.84.254:44955)
0808 15:49:51.644020 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:51.644025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:51.645870 (+ 1845us) server_negotiation.cc:407] Connection header received
0808 15:49:51.648113 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:51.648118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:51.648180 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:51.648320 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:51.649985 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:51.649995 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:51.650003 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:51.650028 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:51.650075 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:51.650097 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:51.650217 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:51.650500 (+ 283us) server_negotiation.cc:299] Negotiation successful
0808 15:49:51.650558 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260808 15:49:51.652315 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:51.644335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44955 (local address 127.27.84.194:43601)
0808 15:49:51.645678 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:51.645743 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:51.647838 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:51.648633 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:51.648645 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:51.648781 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:51.649536 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:49:51.649543 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:51.649605 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:49:51.649731 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:51.650216 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:51.650231 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:51.650252 (+ 21us) client_negotiation.cc:769] Sending connection context
0808 15:49:51.650561 (+ 309us) client_negotiation.cc:240] Negotiation successful
0808 15:49:51.650882 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":327,"threads_started":1}
I20260808 15:49:51.653870 11485 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44955
I20260808 15:49:51.654211 11485 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:51.655002 11485 heartbeater.cc:511] Master 127.27.84.254:44955 requested a full tablet report, sending...
I20260808 15:49:51.656627 10984 ts_manager.cc:194] Registered new tserver with Master: e8fdf019672740eca550838e817691bd (127.27.84.194:36801)
W20260808 15:49:52.005787 11488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:52.006088 11488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:52.006244 11488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:52.013875 11488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:52.013985 11488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:52.014178 11488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:52.023729 11488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:44955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:52.025148 11488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:52.027089 11488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:52.041054 11498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:52.041115 11496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:52.041160 11504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:52.042101 11488 server_base.cc:1034] running on GCE node
I20260808 15:49:52.042816 11488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:52.043766 11488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:52.044978 11488 hybrid_clock.cc:648] HybridClock initialized: now 1786204192044936 us; error 28 us; skew 500 ppm
I20260808 15:49:52.045269 11488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:49:52.047637 11488 webserver.cc:460] Webserver started at http://127.27.84.195:44635/ using document root <none> and password file <none>
I20260808 15:49:52.048323 11488 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:52.048413 11488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:52.048681 11488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:52.051740 11488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "6e32cc38ba4c4266b9c2407a2aba4455"
format_stamp: "Formatted at 2026-08-08 15:49:52 on dist-test-slave-twwt"
server_key: "812a44d139491ad5cd4a0907db22d0e8"
server_key_iv: "4971192ff5883f2cd8e8c1d1f9787acf"
server_key_version: "encryptionkey@0"
I20260808 15:49:52.052487 11488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "6e32cc38ba4c4266b9c2407a2aba4455"
format_stamp: "Formatted at 2026-08-08 15:49:52 on dist-test-slave-twwt"
server_key: "812a44d139491ad5cd4a0907db22d0e8"
server_key_iv: "4971192ff5883f2cd8e8c1d1f9787acf"
server_key_version: "encryptionkey@0"
I20260808 15:49:52.057804 11488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260808 15:49:52.061297 11513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:52.062659 11488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:49:52.062829 11488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "6e32cc38ba4c4266b9c2407a2aba4455"
format_stamp: "Formatted at 2026-08-08 15:49:52 on dist-test-slave-twwt"
server_key: "812a44d139491ad5cd4a0907db22d0e8"
server_key_iv: "4971192ff5883f2cd8e8c1d1f9787acf"
server_key_version: "encryptionkey@0"
I20260808 15:49:52.063109 11488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:52.068804 11488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:52.069581 11488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:52.069851 11488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:52.070873 11488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:52.072270 11488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:52.072367 11488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:52.072448 11488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:52.072503 11488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:52.107187 11488 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:41351
I20260808 15:49:52.107293 11680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:41351 every 8 connection(s)
I20260808 15:49:52.108664 11488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:52.111869 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 11488
I20260808 15:49:52.112056 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:52.112586 27987 external_mini_cluster.cc:1428] Setting key ab006efb136330ffe760232df108fac2
I20260808 15:49:52.117964 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:52.111406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:55621 (local address 127.27.84.254:44955)
0808 15:49:52.111622 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:52.111629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:52.113598 (+ 1969us) server_negotiation.cc:407] Connection header received
0808 15:49:52.115741 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:52.115749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:52.115817 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:52.115925 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:52.117346 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:52.117350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:52.117353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:52.117369 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:52.117402 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:52.117414 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:52.117505 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:52.117688 (+ 183us) server_negotiation.cc:299] Negotiation successful
0808 15:49:52.117738 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:49:52.119064 11683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:52.112145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:44955 (local address 127.27.84.195:55621)
0808 15:49:52.113416 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:52.113475 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:52.115423 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:52.116224 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:52.116233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:52.116332 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:52.116976 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:49:52.116979 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:52.117019 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:49:52.117171 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:52.117509 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:52.117518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:52.117523 (+ 5us) client_negotiation.cc:769] Sending connection context
0808 15:49:52.117742 (+ 219us) client_negotiation.cc:240] Negotiation successful
0808 15:49:52.117947 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":359,"threads_started":1}
I20260808 15:49:52.120333 11681 heartbeater.cc:347] Connected to a master server at 127.27.84.254:44955
I20260808 15:49:52.120639 11681 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:52.121284 11681 heartbeater.cc:511] Master 127.27.84.254:44955 requested a full tablet report, sending...
I20260808 15:49:52.122722 10984 ts_manager.cc:194] Registered new tserver with Master: 6e32cc38ba4c4266b9c2407a2aba4455 (127.27.84.195:41351)
I20260808 15:49:52.128687 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260808 15:49:52.135556 11292 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:49:52.133731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.27.84.254:44955)
0808 15:49:52.134001 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:52.134009 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:52.134454 (+ 445us) server_negotiation.cc:407] Connection header received
0808 15:49:52.134704 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:52.134707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:52.134762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:52.134875 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:52.135249 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43140: BlockingRecv error: recv got EOF from 127.0.0.1:43140 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:49:52.136910 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 11088
I20260808 15:49:52.140823 11695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:52.138054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.27.84.193:35443)
0808 15:49:52.138699 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:52.138707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:52.138735 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:49:52.138809 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:52.138815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:52.139045 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:52.139180 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:52.140160 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:52.140166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:52.140178 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:52.140195 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:52.140236 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:52.140250 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:52.140385 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:52.140481 (+ 96us) server_negotiation.cc:299] Negotiation successful
0808 15:49:52.140567 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":327,"threads_started":1}
I20260808 15:49:52.142263 11280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:52.223656 11287 heartbeater.cc:503] Master 127.27.84.254:44955 was elected leader, sending a full tablet report...
I20260808 15:49:52.224747 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 11088
W20260808 15:49:52.247677 10953 connection.cc:535] server connection from 127.27.84.193:45129 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:49:52.248000 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 11294
I20260808 15:49:52.251931 11699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:52.249168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.27.84.194:36801)
0808 15:49:52.249749 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:52.249757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:52.249788 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:49:52.249855 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:52.249862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:52.250087 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:52.250234 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:52.251168 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:52.251175 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:52.251187 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:52.251218 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:52.251264 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:52.251280 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:52.251392 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:52.251496 (+ 104us) server_negotiation.cc:299] Negotiation successful
0808 15:49:52.251621 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":285,"threads_started":1}
I20260808 15:49:52.253232 11475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:52.337471 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 11294
I20260808 15:49:52.364951 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 11488
I20260808 15:49:52.369345 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:52.366036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50302 (local address 127.27.84.195:41351)
0808 15:49:52.366813 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:52.366825 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:52.366864 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:49:52.366931 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:52.366938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:52.367164 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:52.367343 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:52.368565 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:52.368572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:52.368583 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:49:52.368595 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:52.368639 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:49:52.368653 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:52.368855 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:52.368930 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 15:49:52.369045 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":326,"threads_started":1}
I20260808 15:49:52.370777 11673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:52.453107 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 11488
I20260808 15:49:52.479115 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 10923
I20260808 15:49:52.480492 11035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:52.586208 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 10923
2026-08-08T15:49:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2558 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:52 dist-test-slave-twwt krb5kdc[11708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:52 dist-test-slave-twwt krb5kdc[11708](info): set up 2 sockets
Aug 08 15:49:52 dist-test-slave-twwt krb5kdc[11708](info): commencing operation
krb5kdc: starting...
W20260808 15:49:53.938567 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:54 dist-test-slave-twwt krb5kdc[11708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:49:54.270357 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:33937
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:33937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:54.623773 11729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:54.624084 11729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:54.624250 11729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:54.631583 11729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:49:54.631700 11729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:54.631821 11729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:54.631907 11729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:49:54.631978 11729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:49:54.642104 11729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:33937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:54.643584 11729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:54.645049 11729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:54.657413 11738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:54.657469 11741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:54.657500 11739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:54.658528 11729 server_base.cc:1034] running on GCE node
I20260808 15:49:54.659503 11729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:54.661015 11729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:54.662278 11729 hybrid_clock.cc:648] HybridClock initialized: now 1786204194662219 us; error 60 us; skew 500 ppm
I20260808 15:49:54.662726 11729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:54 dist-test-slave-twwt krb5kdc[11708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204194, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:54.669626 11729 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:49:54.671620 11729 webserver.cc:460] Webserver started at http://127.27.84.254:34659/ using document root <none> and password file <none>
I20260808 15:49:54.672470 11729 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:54.672644 11729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:54.673048 11729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:54.676542 11729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "27cc367173414ad2a359355dddcddea2"
format_stamp: "Formatted at 2026-08-08 15:49:54 on dist-test-slave-twwt"
server_key: "1aa47d4ae817fc63118917c418d44249"
server_key_iv: "00a764204b1c2080650329432f59daa2"
server_key_version: "encryptionkey@0"
I20260808 15:49:54.677549 11729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "27cc367173414ad2a359355dddcddea2"
format_stamp: "Formatted at 2026-08-08 15:49:54 on dist-test-slave-twwt"
server_key: "1aa47d4ae817fc63118917c418d44249"
server_key_iv: "00a764204b1c2080650329432f59daa2"
server_key_version: "encryptionkey@0"
I20260808 15:49:54.683717 11729 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260808 15:49:54.687673 11753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:54.689188 11729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:49:54.689409 11729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "27cc367173414ad2a359355dddcddea2"
format_stamp: "Formatted at 2026-08-08 15:49:54 on dist-test-slave-twwt"
server_key: "1aa47d4ae817fc63118917c418d44249"
server_key_iv: "00a764204b1c2080650329432f59daa2"
server_key_version: "encryptionkey@0"
I20260808 15:49:54.689651 11729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:54.712163 11729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:54.717231 11729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:54.717510 11729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:54.737321 11729 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:33937
I20260808 15:49:54.737365 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:33937 every 8 connection(s)
I20260808 15:49:54.738963 11729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:49:54.741089 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 11729
I20260808 15:49:54.741369 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:49:54.742256 27987 external_mini_cluster.cc:1428] Setting key 308e5760c23dd6493ba33dee32fe6863
I20260808 15:49:54.744072 11833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:49:54 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:54.759003 11833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2: Bootstrap starting.
I20260808 15:49:54.762656 11833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:54.763727 11833 log.cc:824] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:54.766736 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:54.744561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.27.84.254:33937)
0808 15:49:54.745352 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:54.745364 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:54.745405 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:49:54.747121 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:54.747143 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:54.747561 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:54.747911 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:54.749940 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:54.751494 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:54.753343 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:54.754028 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:54.754165 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:54.759088 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:54.759096 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:54.759111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:54.759185 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:54.762323 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:54.763185 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:54.763193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:54.763199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:54.763387 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:54.763791 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:54.763797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:54.763800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:54.764431 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:54.764796 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:54.765234 (+ 438us) server_negotiation.cc:299] Negotiation successful
0808 15:49:54.765684 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":363,"threads_started":1}
I20260808 15:49:54.766983 11833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2: No bootstrap required, opened a new log
I20260808 15:49:54.775674 11833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc367173414ad2a359355dddcddea2" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33937 } }
I20260808 15:49:54.776101 11833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:54.776221 11833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cc367173414ad2a359355dddcddea2, State: Initialized, Role: FOLLOWER
I20260808 15:49:54.776922 11833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc367173414ad2a359355dddcddea2" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33937 } }
I20260808 15:49:54.777181 11833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:49:54.777339 11833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:49:54.777554 11833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:54.780344 11833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc367173414ad2a359355dddcddea2" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33937 } }
I20260808 15:49:54.780869 11833 leader_election.cc:302] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27cc367173414ad2a359355dddcddea2; no voters:
I20260808 15:49:54.781526 11833 leader_election.cc:288] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:49:54.781682 11843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:54.782431 11843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [term 1 LEADER]: Becoming Leader. State: Replica: 27cc367173414ad2a359355dddcddea2, State: Running, Role: LEADER
I20260808 15:49:54.783075 11843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc367173414ad2a359355dddcddea2" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33937 } }
I20260808 15:49:54.783711 11833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:49:54.785974 11845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27cc367173414ad2a359355dddcddea2. Latest consensus state: current_term: 1 leader_uuid: "27cc367173414ad2a359355dddcddea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc367173414ad2a359355dddcddea2" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33937 } } }
I20260808 15:49:54.786321 11845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:54.786883 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27cc367173414ad2a359355dddcddea2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27cc367173414ad2a359355dddcddea2" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 33937 } } }
I20260808 15:49:54.787273 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2 [sys.catalog]: This master's current role is: LEADER
I20260808 15:49:54.788791 11851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:49:54.794274 11851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:49:54.803941 11851 catalog_manager.cc:1347] Generated new cluster ID: 287d1e551d444204a305b2c46a64e198
I20260808 15:49:54.804075 11851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:49:54.828040 11851 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:49:54.829401 11851 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:49:54.838898 11851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2: Generated new TSK 0
I20260808 15:49:54.839883 11851 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:49:54.918182 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33937
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:55.310654 11873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:55.310994 11873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:55.311180 11873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:55.318523 11873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:55.318701 11873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:55.318931 11873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:49:55.329232 11873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:55.330718 11873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:55.332147 11873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:55.346791 11885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:55.346832 11881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:55.346705 11882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:55.352280 11873 server_base.cc:1034] running on GCE node
I20260808 15:49:55.352968 11873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:55.353935 11873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:55.355211 11873 hybrid_clock.cc:648] HybridClock initialized: now 1786204195355123 us; error 59 us; skew 500 ppm
I20260808 15:49:55.355608 11873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:55 dist-test-slave-twwt krb5kdc[11708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:55.359062 11873 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:49:55.360628 11873 webserver.cc:460] Webserver started at http://127.27.84.193:43543/ using document root <none> and password file <none>
I20260808 15:49:55.361370 11873 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:55.361476 11873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:55.361732 11873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:55.364867 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "fed87e65f38d45e8ab8c93926b3aa2dd"
format_stamp: "Formatted at 2026-08-08 15:49:55 on dist-test-slave-twwt"
server_key: "9e263433fc0e8720be80255dc0e1e520"
server_key_iv: "fe8d2fd59594b73dcbcc887f3c569e46"
server_key_version: "encryptionkey@0"
I20260808 15:49:55.365717 11873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "fed87e65f38d45e8ab8c93926b3aa2dd"
format_stamp: "Formatted at 2026-08-08 15:49:55 on dist-test-slave-twwt"
server_key: "9e263433fc0e8720be80255dc0e1e520"
server_key_iv: "fe8d2fd59594b73dcbcc887f3c569e46"
server_key_version: "encryptionkey@0"
I20260808 15:49:55.371204 11873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 15:49:55.374894 11897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:55.376405 11873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:49:55.376600 11873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "fed87e65f38d45e8ab8c93926b3aa2dd"
format_stamp: "Formatted at 2026-08-08 15:49:55 on dist-test-slave-twwt"
server_key: "9e263433fc0e8720be80255dc0e1e520"
server_key_iv: "fe8d2fd59594b73dcbcc887f3c569e46"
server_key_version: "encryptionkey@0"
I20260808 15:49:55.376822 11873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:55.396327 11873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:55.400612 11873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:55.400959 11873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:55.401952 11873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:55.403404 11873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:55.403512 11873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:55.403669 11873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:55.403752 11873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:55.439659 11873 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:39557
I20260808 15:49:55.439708 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:39557 every 8 connection(s)
I20260808 15:49:55.441069 11873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:49:55.447237 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 11873
I20260808 15:49:55.447502 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:49:55.448220 27987 external_mini_cluster.cc:1428] Setting key b40c1e19d624ad0a94aa0f77eacbcf0a
Aug 08 15:49:55 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:55.464720 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:55.443845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:40077 (local address 127.27.84.254:33937)
0808 15:49:55.444326 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:55.444334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:55.445427 (+ 1093us) server_negotiation.cc:407] Connection header received
0808 15:49:55.447386 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:55.447393 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:55.447507 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:55.447638 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:55.449452 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:55.450507 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:55.453179 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:55.453586 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:55.453699 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:55.458253 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:55.458259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:55.458262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:55.458298 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:55.461228 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:55.462061 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:55.462069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:55.462072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:55.462222 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:55.462772 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:55.462776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:55.462778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:55.463099 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:55.463263 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:55.464038 (+ 775us) server_negotiation.cc:299] Negotiation successful
0808 15:49:55.464314 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260808 15:49:55.465786 12086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:55.444461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33937 (local address 127.27.84.193:40077)
0808 15:49:55.445259 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:55.445301 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:55.447073 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:55.447918 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:55.447927 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:55.448533 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:55.449258 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:55.449288 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:55.450737 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:55.450742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:55.452975 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:55.452997 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:55.453851 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:55.453860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:55.454222 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:55.455120 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:55.455163 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:55.457982 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:55.461428 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:55.461438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:55.461464 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:55.461920 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:55.462348 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:55.462353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:55.462357 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:55.462567 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:55.463238 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:55.463247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:55.463630 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 15:49:55.463924 (+ 294us) client_negotiation.cc:240] Negotiation successful
0808 15:49:55.464590 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":318,"threads_started":1}
I20260808 15:49:55.467518 12083 heartbeater.cc:347] Connected to a master server at 127.27.84.254:33937
I20260808 15:49:55.467875 12083 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:55.468904 12083 heartbeater.cc:511] Master 127.27.84.254:33937 requested a full tablet report, sending...
I20260808 15:49:55.471496 11783 ts_manager.cc:194] Registered new tserver with Master: fed87e65f38d45e8ab8c93926b3aa2dd (127.27.84.193:39557)
I20260808 15:49:55.473776 11783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:40077
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:49:55.524384 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33937
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:55.895248 12089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:55.895558 12089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:55.895720 12089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:55.902973 12089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:55.903095 12089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:55.903254 12089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:49:55.913415 12089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:55.914866 12089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:55.916234 12089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:55.930742 12097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:55.930800 12098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:55.930827 12102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:55.931871 12089 server_base.cc:1034] running on GCE node
I20260808 15:49:55.932559 12089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:55.933597 12089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:55.934855 12089 hybrid_clock.cc:648] HybridClock initialized: now 1786204195934774 us; error 67 us; skew 500 ppm
I20260808 15:49:55.935223 12089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:55 dist-test-slave-twwt krb5kdc[11708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:55.938934 12089 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:49:55.940580 12089 webserver.cc:460] Webserver started at http://127.27.84.194:38283/ using document root <none> and password file <none>
I20260808 15:49:55.941351 12089 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:55.941489 12089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:55.941828 12089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:55.945591 12089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "724cc6f7ca78443f9b96c33f8cd604c8"
format_stamp: "Formatted at 2026-08-08 15:49:55 on dist-test-slave-twwt"
server_key: "d075f17650a5479ec7d656b3257b9c82"
server_key_iv: "de91792198efc5240dcf766d71403e55"
server_key_version: "encryptionkey@0"
I20260808 15:49:55.946437 12089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "724cc6f7ca78443f9b96c33f8cd604c8"
format_stamp: "Formatted at 2026-08-08 15:49:55 on dist-test-slave-twwt"
server_key: "d075f17650a5479ec7d656b3257b9c82"
server_key_iv: "de91792198efc5240dcf766d71403e55"
server_key_version: "encryptionkey@0"
I20260808 15:49:55.953328 12089 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 15:49:55.958766 12117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:55.960492 12089 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:49:55.960652 12089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "724cc6f7ca78443f9b96c33f8cd604c8"
format_stamp: "Formatted at 2026-08-08 15:49:55 on dist-test-slave-twwt"
server_key: "d075f17650a5479ec7d656b3257b9c82"
server_key_iv: "de91792198efc5240dcf766d71403e55"
server_key_version: "encryptionkey@0"
I20260808 15:49:55.960850 12089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:55.987653 12089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:55.993557 12089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:55.993896 12089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:55.995064 12089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:55.996672 12089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:55.996757 12089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:55.996937 12089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:55.997012 12089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:56.032132 12089 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:36565
I20260808 15:49:56.032269 12296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:36565 every 8 connection(s)
I20260808 15:49:56.033639 12089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:49:56.038695 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 12089
I20260808 15:49:56.038928 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:49:56.039379 27987 external_mini_cluster.cc:1428] Setting key fa5fdb5c7a8f6db4edfc7c990f51b6a8
Aug 08 15:49:56 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204195, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:56.058322 12304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.036658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:38301 (local address 127.27.84.254:33937)
0808 15:49:56.037195 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.037200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.038343 (+ 1143us) server_negotiation.cc:407] Connection header received
0808 15:49:56.040856 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.040860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.040947 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.041047 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:56.042815 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.044071 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.045726 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.046507 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.046684 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.050643 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:56.050653 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:56.050662 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:56.050714 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:56.054728 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.055709 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.055713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.055715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.055856 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.056414 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.056418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.056434 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.056715 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:56.056839 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.057823 (+ 984us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.058010 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":235,"threads_started":1}
I20260808 15:49:56.060113 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.037309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33937 (local address 127.27.84.194:38301)
0808 15:49:56.038123 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.038166 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.040517 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.041363 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.041372 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.041958 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:56.042567 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.042587 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.044262 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.044266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.045560 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.045574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.046788 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.046793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.047019 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.047781 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:56.047808 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:56.050346 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:56.054896 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:56.054917 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:56.054933 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:56.055542 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:56.055981 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:56.055989 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:56.055994 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:56.056291 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:56.056860 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:56.056872 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:56.057395 (+ 523us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.057738 (+ 343us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.058410 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":330,"threads_started":1}
I20260808 15:49:56.061795 12298 heartbeater.cc:347] Connected to a master server at 127.27.84.254:33937
I20260808 15:49:56.062150 12298 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:56.063010 12298 heartbeater.cc:511] Master 127.27.84.254:33937 requested a full tablet report, sending...
I20260808 15:49:56.064648 11783 ts_manager.cc:194] Registered new tserver with Master: 724cc6f7ca78443f9b96c33f8cd604c8 (127.27.84.194:36565)
I20260808 15:49:56.065567 11783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:38301
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:49:56.112062 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33937
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:49:56.468971 12314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:49:56.469337 12314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:49:56.469520 12314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:49:56.476946 12314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:49:56.477120 12314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:49:56.477320 12314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:49:56.477913 12083 heartbeater.cc:503] Master 127.27.84.254:33937 was elected leader, sending a full tablet report...
I20260808 15:49:56.488050 12314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:33937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:49:56.489534 12314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:49:56.490958 12314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:49:56.505059 12323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:56.504988 12326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:49:56.505071 12328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:49:56.505724 12314 server_base.cc:1034] running on GCE node
I20260808 15:49:56.506479 12314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:49:56.507555 12314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:49:56.508795 12314 hybrid_clock.cc:648] HybridClock initialized: now 1786204196508716 us; error 63 us; skew 500 ppm
I20260808 15:49:56.509135 12314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:49:56 dist-test-slave-twwt krb5kdc[11708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204196, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:49:56.512645 12314 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:49:56.514223 12314 webserver.cc:460] Webserver started at http://127.27.84.195:34511/ using document root <none> and password file <none>
I20260808 15:49:56.515046 12314 fs_manager.cc:359] Metadata directory not provided
I20260808 15:49:56.515187 12314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:49:56.515527 12314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:49:56.518515 12314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "e842bb5c7ed547c3a72fc0d9fe551f40"
format_stamp: "Formatted at 2026-08-08 15:49:56 on dist-test-slave-twwt"
server_key: "59158d13809ee03c4040b76afb91c91c"
server_key_iv: "f2c5dca0fa67f3ab24d39031b9d6d27a"
server_key_version: "encryptionkey@0"
I20260808 15:49:56.519390 12314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "e842bb5c7ed547c3a72fc0d9fe551f40"
format_stamp: "Formatted at 2026-08-08 15:49:56 on dist-test-slave-twwt"
server_key: "59158d13809ee03c4040b76afb91c91c"
server_key_iv: "f2c5dca0fa67f3ab24d39031b9d6d27a"
server_key_version: "encryptionkey@0"
I20260808 15:49:56.524686 12314 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:49:56.528278 12337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:56.529714 12314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:49:56.529922 12314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "e842bb5c7ed547c3a72fc0d9fe551f40"
format_stamp: "Formatted at 2026-08-08 15:49:56 on dist-test-slave-twwt"
server_key: "59158d13809ee03c4040b76afb91c91c"
server_key_iv: "f2c5dca0fa67f3ab24d39031b9d6d27a"
server_key_version: "encryptionkey@0"
I20260808 15:49:56.530138 12314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:49:56.555624 12314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:49:56.560335 12314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:49:56.560665 12314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:49:56.561843 12314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:49:56.563391 12314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:49:56.563501 12314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:56.563655 12314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:49:56.563738 12314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:49:56.599087 12314 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:38977
I20260808 15:49:56.599144 12527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:38977 every 8 connection(s)
I20260808 15:49:56.600466 12314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:49:56.603901 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 12314
I20260808 15:49:56.604136 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:49:56.604593 27987 external_mini_cluster.cc:1428] Setting key 733fa739aab4ca166a6a9d40d1bbe336
Aug 08 15:49:56 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204196, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:49:56.621493 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.603403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:47429 (local address 127.27.84.254:33937)
0808 15:49:56.604000 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.604008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.605230 (+ 1222us) server_negotiation.cc:407] Connection header received
0808 15:49:56.607173 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.607179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.607267 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.607388 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:56.609029 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.609827 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.611430 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.611811 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.611907 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.615417 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:56.615428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:56.615431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:56.615464 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:56.618043 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.618955 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.618982 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.618988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.619157 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.619681 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.619685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.619687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.619976 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:56.620151 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.620860 (+ 709us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.621082 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":262,"threads_started":1}
I20260808 15:49:56.622532 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.604057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:33937 (local address 127.27.84.195:47429)
0808 15:49:56.605008 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.605083 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.606911 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.607663 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.607670 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.608298 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:49:56.608865 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.608882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.609950 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.609954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.611294 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.611306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.612000 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.612005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.612191 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.612982 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:49:56.613004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:49:56.615212 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:49:56.618192 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:56.618202 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:56.618218 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:56.618828 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:56.619278 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:49:56.619283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:49:56.619286 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:49:56.619552 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:49:56.620123 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:49:56.620131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:49:56.620478 (+ 347us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.620763 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.621324 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":393,"threads_started":1}
I20260808 15:49:56.623937 12528 heartbeater.cc:347] Connected to a master server at 127.27.84.254:33937
I20260808 15:49:56.624226 12528 heartbeater.cc:464] Registering TS with master...
I20260808 15:49:56.624933 12528 heartbeater.cc:511] Master 127.27.84.254:33937 requested a full tablet report, sending...
I20260808 15:49:56.626341 11783 ts_manager.cc:194] Registered new tserver with Master: e842bb5c7ed547c3a72fc0d9fe551f40 (127.27.84.195:38977)
I20260808 15:49:56.627184 11783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:47429
I20260808 15:49:56.632818 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:49:56.650350 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.637738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.27.84.254:33937)
0808 15:49:56.637917 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.637929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.638521 (+ 592us) server_negotiation.cc:407] Connection header received
0808 15:49:56.638772 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.638776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.638841 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.638946 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:56.640424 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.641183 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.642416 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.642888 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.642994 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.645341 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:56.645346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:56.645349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:56.645380 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:56.647961 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.648666 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.648670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.648673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.648789 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.649128 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.649131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.649133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.649467 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:56.649643 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.649940 (+ 297us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.650112 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:49:56.654500 11783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:49:56.661507 11783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:49:56.687351 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.674134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:36565 (local address 127.0.0.1:49740)
0808 15:49:56.675254 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.675324 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.675545 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.677004 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.677016 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.677060 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:56.677590 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.677608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.680516 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.680525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.683726 (+ 3201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.683750 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.686258 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.686268 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.686490 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.686548 (+ 58us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.686696 (+ 148us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.686897 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":592,"threads_started":1}
I20260808 15:49:56.688537 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.674408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49740 (local address 127.27.84.194:36565)
0808 15:49:56.676196 (+ 1788us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.676209 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.676246 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:49:56.676382 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.676393 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.676707 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.677853 (+ 1146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:56.678169 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.680313 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.684023 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.685969 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.687692 (+ 1723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.687914 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.687981 (+ 67us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.688159 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1609,"thread_start_us":1489,"threads_started":1}
I20260808 15:49:56.693094 12189 tablet_service.cc:1467] Processing CreateTablet for tablet ad9080b603684efd85b4d4ec16619ec9 (DEFAULT_TABLE table=test-table [id=e873ea511c1f43dbac28d7207419499e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:56.694936 12189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9080b603684efd85b4d4ec16619ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:56.696220 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.674871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:38977 (local address 127.0.0.1:38782)
0808 15:49:56.687267 (+ 12396us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.687285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.687422 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.688485 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.688491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.688518 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:56.689641 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.689656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.691453 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.691460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.693423 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.693441 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.695332 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.695339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.695506 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.695543 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.695647 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.695792 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12284,"mutex_wait_us":899,"thread_start_us":8832,"threads_started":1}
I20260808 15:49:56.696247 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.675257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38782 (local address 127.27.84.195:38977)
0808 15:49:56.686627 (+ 11370us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.686636 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.687564 (+ 928us) server_negotiation.cc:407] Connection header received
0808 15:49:56.687680 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.687687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.687993 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.688190 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:56.689856 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.691307 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.693618 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.695105 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.695253 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.695511 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.695710 (+ 199us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.695889 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11172,"thread_start_us":11393,"threads_started":1}
I20260808 15:49:56.696647 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.675545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45072)
0808 15:49:56.684879 (+ 9334us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.684901 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.685018 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.688343 (+ 3325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.688348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.688376 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:56.688801 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.688815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.690807 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.690812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.693112 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.693131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.695769 (+ 2638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.695776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.695964 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.696000 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.696100 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.696246 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9183,"mutex_wait_us":56,"thread_start_us":11455,"threads_started":1}
I20260808 15:49:56.697556 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.676126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.27.84.193:39557)
0808 15:49:56.687694 (+ 11568us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.687704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.687734 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:49:56.687834 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.687841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.688111 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.688327 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:56.689010 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.690650 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.693584 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.695230 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.695352 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.695565 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.697029 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.697205 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11386,"thread_start_us":11219,"threads_started":1}
I20260808 15:49:56.700598 12420 tablet_service.cc:1467] Processing CreateTablet for tablet ad9080b603684efd85b4d4ec16619ec9 (DEFAULT_TABLE table=test-table [id=e873ea511c1f43dbac28d7207419499e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:56.701864 12420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9080b603684efd85b4d4ec16619ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:56.702201 11978 tablet_service.cc:1467] Processing CreateTablet for tablet ad9080b603684efd85b4d4ec16619ec9 (DEFAULT_TABLE table=test-table [id=e873ea511c1f43dbac28d7207419499e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:49:56.704203 11978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad9080b603684efd85b4d4ec16619ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:49:56.710548 12556 tablet_bootstrap.cc:492] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Bootstrap starting.
I20260808 15:49:56.714764 12556 tablet_bootstrap.cc:654] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:56.715377 12557 tablet_bootstrap.cc:492] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Bootstrap starting.
I20260808 15:49:56.716742 12556 log.cc:824] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:56.719388 12561 tablet_bootstrap.cc:492] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: Bootstrap starting.
I20260808 15:49:56.719652 12557 tablet_bootstrap.cc:654] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:56.720646 12556 tablet_bootstrap.cc:492] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: No bootstrap required, opened a new log
I20260808 15:49:56.720846 12557 log.cc:824] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:56.720961 12556 ts_tablet_manager.cc:1397] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260808 15:49:56.723188 12561 tablet_bootstrap.cc:654] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: Neither blocks nor log segments found. Creating new log.
I20260808 15:49:56.723655 12557 tablet_bootstrap.cc:492] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: No bootstrap required, opened a new log
I20260808 15:49:56.723935 12557 ts_tablet_manager.cc:1397] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260808 15:49:56.724454 12561 log.cc:824] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: Log is configured to *not* fsync() on all Append() calls
I20260808 15:49:56.729377 12561 tablet_bootstrap.cc:492] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: No bootstrap required, opened a new log
I20260808 15:49:56.729667 12561 ts_tablet_manager.cc:1397] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260808 15:49:56.732050 12557 raft_consensus.cc:348] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.732445 12557 raft_consensus.cc:374] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:56.732610 12557 raft_consensus.cc:729] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e842bb5c7ed547c3a72fc0d9fe551f40, State: Initialized, Role: FOLLOWER
I20260808 15:49:56.732474 12556 raft_consensus.cc:348] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.732961 12556 raft_consensus.cc:374] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:56.733064 12556 raft_consensus.cc:729] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 724cc6f7ca78443f9b96c33f8cd604c8, State: Initialized, Role: FOLLOWER
I20260808 15:49:56.733456 12557 consensus_queue.cc:260] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.733678 12556 consensus_queue.cc:260] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.735121 12298 heartbeater.cc:503] Master 127.27.84.254:33937 was elected leader, sending a full tablet report...
I20260808 15:49:56.735824 12556 ts_tablet_manager.cc:1428] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260808 15:49:56.736820 12528 heartbeater.cc:503] Master 127.27.84.254:33937 was elected leader, sending a full tablet report...
I20260808 15:49:56.737515 12557 ts_tablet_manager.cc:1428] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260808 15:49:56.741520 12561 raft_consensus.cc:348] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.742103 12561 raft_consensus.cc:374] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:49:56.742264 12561 raft_consensus.cc:729] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed87e65f38d45e8ab8c93926b3aa2dd, State: Initialized, Role: FOLLOWER
I20260808 15:49:56.743134 12561 consensus_queue.cc:260] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.745154 12561 ts_tablet_manager.cc:1428] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260808 15:49:56.785866 12303 tablet.cc:2369] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:56.788450 12564 raft_consensus.cc:482] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:49:56.788834 12564 raft_consensus.cc:504] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.790654 12564 leader_election.cc:288] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 724cc6f7ca78443f9b96c33f8cd604c8 (127.27.84.194:36565), fed87e65f38d45e8ab8c93926b3aa2dd (127.27.84.193:39557)
I20260808 15:49:56.797607 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.791054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:54699 (local address 127.27.84.194:36565)
0808 15:49:56.791254 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.791258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.791387 (+ 129us) server_negotiation.cc:407] Connection header received
0808 15:49:56.791608 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.791636 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.791729 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.791860 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:56.792518 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.793578 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.795601 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.796778 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.796901 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.796953 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.797314 (+ 361us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.797407 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:49:56.797724 12533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.791080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:36565 (local address 127.27.84.195:54699)
0808 15:49:56.791283 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.791313 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.791453 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.792109 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.792113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.792140 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:56.792307 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.792317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.793749 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.793755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.795415 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.795435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.796935 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.796942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.797139 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.797181 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.797283 (+ 102us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.797439 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:49:56.798722 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.791804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:57979 (local address 127.27.84.193:39557)
0808 15:49:56.792023 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.792031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.792380 (+ 349us) server_negotiation.cc:407] Connection header received
0808 15:49:56.792552 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.792556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.792613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.792681 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:49:56.793338 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.794411 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.796548 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.797689 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.797861 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.797949 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.798368 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.798495 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:49:56.798813 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.791500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.27.84.195:57979)
0808 15:49:56.792264 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:56.792278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0808 15:49:56.792390 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:49:56.792760 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:49:56.792766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:49:56.792793 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:49:56.793142 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.793154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.794618 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.794622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.796388 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:49:56.796403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.798048 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.798058 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:49:56.798210 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.798233 (+ 23us) client_negotiation.cc:769] Sending connection context
0808 15:49:56.798309 (+ 76us) client_negotiation.cc:240] Negotiation successful
0808 15:49:56.798454 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":476,"threads_started":1}
I20260808 15:49:56.799005 12223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9080b603684efd85b4d4ec16619ec9" candidate_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" is_pre_election: true
I20260808 15:49:56.799649 12223 raft_consensus.cc:2454] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e842bb5c7ed547c3a72fc0d9fe551f40 in term 0.
I20260808 15:49:56.799975 12016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9080b603684efd85b4d4ec16619ec9" candidate_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" is_pre_election: true
I20260808 15:49:56.800364 12344 leader_election.cc:302] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 724cc6f7ca78443f9b96c33f8cd604c8, e842bb5c7ed547c3a72fc0d9fe551f40; no voters:
I20260808 15:49:56.800588 12016 raft_consensus.cc:2454] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e842bb5c7ed547c3a72fc0d9fe551f40 in term 0.
I20260808 15:49:56.800791 12564 raft_consensus.cc:2781] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:49:56.800948 12564 raft_consensus.cc:482] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:49:56.801092 12564 raft_consensus.cc:3037] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:56.803498 12564 raft_consensus.cc:504] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.804149 12564 leader_election.cc:288] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [CANDIDATE]: Term 1 election: Requested vote from peers 724cc6f7ca78443f9b96c33f8cd604c8 (127.27.84.194:36565), fed87e65f38d45e8ab8c93926b3aa2dd (127.27.84.193:39557)
I20260808 15:49:56.804662 12223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9080b603684efd85b4d4ec16619ec9" candidate_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "724cc6f7ca78443f9b96c33f8cd604c8"
I20260808 15:49:56.804718 12016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad9080b603684efd85b4d4ec16619ec9" candidate_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd"
I20260808 15:49:56.804888 12223 raft_consensus.cc:3037] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:56.804912 12016 raft_consensus.cc:3037] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:49:56.807191 12016 raft_consensus.cc:2454] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e842bb5c7ed547c3a72fc0d9fe551f40 in term 1.
I20260808 15:49:56.807204 12223 raft_consensus.cc:2454] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e842bb5c7ed547c3a72fc0d9fe551f40 in term 1.
I20260808 15:49:56.807607 12344 leader_election.cc:302] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e842bb5c7ed547c3a72fc0d9fe551f40, fed87e65f38d45e8ab8c93926b3aa2dd; no voters:
I20260808 15:49:56.807982 12564 raft_consensus.cc:2781] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:49:56.808588 12564 raft_consensus.cc:686] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 1 LEADER]: Becoming Leader. State: Replica: e842bb5c7ed547c3a72fc0d9fe551f40, State: Running, Role: LEADER
I20260808 15:49:56.809114 12564 consensus_queue.cc:237] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } }
I20260808 15:49:56.814247 11780 catalog_manager.cc:5495] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 reported cstate change: term changed from 0 to 1, leader changed from <none> to e842bb5c7ed547c3a72fc0d9fe551f40 (127.27.84.195). New cstate: current_term: 1 leader_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e842bb5c7ed547c3a72fc0d9fe551f40" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 38977 } health_report { overall_health: HEALTHY } } }
W20260808 15:49:56.852141 12529 tablet.cc:2369] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:49:56.857468 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.851592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.27.84.195:38977)
0808 15:49:56.851838 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.851845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.852001 (+ 156us) server_negotiation.cc:407] Connection header received
0808 15:49:56.852286 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.852291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.852421 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.852582 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:49:56.852979 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.853734 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.855132 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.855574 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.855736 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.856279 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:49:56.856847 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:49:56.857021 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.857080 (+ 59us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.857168 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:49:56.866317 12016 raft_consensus.cc:1261] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 1 FOLLOWER]: Refusing update from remote peer e842bb5c7ed547c3a72fc0d9fe551f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:56.866322 12223 raft_consensus.cc:1261] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 1 FOLLOWER]: Refusing update from remote peer e842bb5c7ed547c3a72fc0d9fe551f40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:49:56.867403 12569 consensus_queue.cc:1035] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "724cc6f7ca78443f9b96c33f8cd604c8" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:56.867766 12564 consensus_queue.cc:1035] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fed87e65f38d45e8ab8c93926b3aa2dd" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:49:56.882143 12577 mvcc.cc:205] Tried to move back new op lower bound from 7316292390353436672 to 7316292390134726656. Current Snapshot: MvccSnapshot[applied={T|T < 7316292390353436672 or (T in {7316292390353436672})}]
I20260808 15:49:56.883805 12575 mvcc.cc:205] Tried to move back new op lower bound from 7316292390353436672 to 7316292390134726656. Current Snapshot: MvccSnapshot[applied={T|T < 7316292390353436672}]
I20260808 15:49:56.896554 11781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:56.896883 11781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:49:56.900859 11781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27cc367173414ad2a359355dddcddea2: Sending DeleteTablet for 3 replicas of tablet ad9080b603684efd85b4d4ec16619ec9
I20260808 15:49:56.902073 12189 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9080b603684efd85b4d4ec16619ec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:56 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:49740
I20260808 15:49:56.902132 11978 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9080b603684efd85b4d4ec16619ec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:56 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45072
I20260808 15:49:56.902364 12420 tablet_service.cc:1514] Processing DeleteTablet for tablet ad9080b603684efd85b4d4ec16619ec9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:49:56 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:38782
I20260808 15:49:56.904836 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 11873
I20260808 15:49:56.919260 12594 tablet_replica.cc:330] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: stopping tablet replica
I20260808 15:49:56.919924 12594 raft_consensus.cc:2229] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:49:56.920300 12590 tablet_replica.cc:330] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: stopping tablet replica
I20260808 15:49:56.920782 12590 raft_consensus.cc:2229] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:56.920820 12594 raft_consensus.cc:2258] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:56.921370 12590 raft_consensus.cc:2258] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:56.923440 12594 ts_tablet_manager.cc:1905] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:56.923532 12590 ts_tablet_manager.cc:1905] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:56.930939 12590 ts_tablet_manager.cc:1918] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:56.931134 12590 log.cc:1196] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/ad9080b603684efd85b4d4ec16619ec9
I20260808 15:49:56.931712 12590 ts_tablet_manager.cc:1939] T ad9080b603684efd85b4d4ec16619ec9 P 724cc6f7ca78443f9b96c33f8cd604c8: Deleting consensus metadata
Aug 08 15:49:56 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:49:56.933892 12592 tablet_replica.cc:330] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: stopping tablet replica
I20260808 15:49:56.934402 12592 raft_consensus.cc:2229] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:49:56.935031 12592 raft_consensus.cc:2258] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:49:56.930999 12594 ts_tablet_manager.cc:1918] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:49:56.935277 11758 catalog_manager.cc:4841] TS 724cc6f7ca78443f9b96c33f8cd604c8 (127.27.84.194:36565): tablet ad9080b603684efd85b4d4ec16619ec9 (table test-table [id=e873ea511c1f43dbac28d7207419499e]) successfully deleted
I20260808 15:49:56.937542 12592 ts_tablet_manager.cc:1905] T ad9080b603684efd85b4d4ec16619ec9 P fed87e65f38d45e8ab8c93926b3aa2dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:49:56.937772 12594 log.cc:1196] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/ad9080b603684efd85b4d4ec16619ec9
I20260808 15:49:56.938405 12594 ts_tablet_manager.cc:1939] T ad9080b603684efd85b4d4ec16619ec9 P e842bb5c7ed547c3a72fc0d9fe551f40: Deleting consensus metadata
I20260808 15:49:56.940325 11754 catalog_manager.cc:4841] TS e842bb5c7ed547c3a72fc0d9fe551f40 (127.27.84.195:38977): tablet ad9080b603684efd85b4d4ec16619ec9 (table test-table [id=e873ea511c1f43dbac28d7207419499e]) successfully deleted
I20260808 15:49:56.943145 12555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:56.906576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.27.84.193:39557)
0808 15:49:56.906844 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:56.906854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:56.919595 (+ 12741us) server_negotiation.cc:407] Connection header received
0808 15:49:56.919834 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:56.919843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:56.919969 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:56.920098 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:56.922843 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.924469 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.927153 (+ 2684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:56.928155 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:56.928307 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:56.935878 (+ 7571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:56.935892 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:56.935906 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:56.935946 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:56.939521 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.940461 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.940471 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.940475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.940654 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:56.941158 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:56.941164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:56.941167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:56.941637 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:56.941943 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:56.942409 (+ 466us) server_negotiation.cc:299] Negotiation successful
0808 15:49:56.942701 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260808 15:49:56.943861 12074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:57.060097 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 11873
I20260808 15:49:57.096297 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 12089
W20260808 15:49:57.096225 11758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:39557 (error 108)
I20260808 15:49:57.098671 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.098132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45084)
0808 15:49:57.098347 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.098462 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260808 15:49:57.098973 11758 catalog_manager.cc:4584] TS fed87e65f38d45e8ab8c93926b3aa2dd (127.27.84.193:39557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad9080b603684efd85b4d4ec16619ec9: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Aug 08 15:49:57 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:49:57.113194 12548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.097743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49750 (local address 127.27.84.194:36565)
0808 15:49:57.097980 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:57.097989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:57.098010 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:49:57.098080 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:57.098086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:57.098184 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:57.098290 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:57.099874 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:57.100618 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:57.101887 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:57.102286 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:57.102424 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:57.107082 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:57.107098 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:57.107115 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:57.107149 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:57.110135 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:57.110915 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:57.110930 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:57.110932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:57.111060 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:57.111505 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:57.111509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:57.111511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:57.111892 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:57.112224 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:57.112572 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 15:49:57.112825 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:49:57.113879 12290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:57.152793 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.152175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45086)
0808 15:49:57.152445 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.152575 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 15:49:57.154018 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.153571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45098)
0808 15:49:57.153753 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.153863 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:49:57.211261 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 12089
I20260808 15:49:57.219501 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.218937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45114)
0808 15:49:57.219181 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.219293 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:49:57.220794 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.220302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45126)
0808 15:49:57.220539 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.220628 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260808 15:49:57.238379 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 12314
Aug 08 15:49:57 dist-test-slave-twwt krb5kdc[11708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:49:57.258185 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.239261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.27.84.195:38977)
0808 15:49:57.239440 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:49:57.239445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:49:57.239526 (+ 81us) server_negotiation.cc:407] Connection header received
0808 15:49:57.239732 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:49:57.239735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:49:57.239830 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:49:57.239913 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:49:57.241817 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:57.242771 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:57.244686 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:49:57.245164 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:49:57.245263 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:49:57.251366 (+ 6103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:49:57.251383 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:49:57.251399 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:49:57.251451 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:49:57.254842 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:57.255772 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:57.255779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:57.255782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:57.255927 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:49:57.256653 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:49:57.256656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:49:57.256658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:49:57.257053 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:49:57.257315 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 15:49:57.257655 (+ 340us) server_negotiation.cc:299] Negotiation successful
0808 15:49:57.257822 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:49:57.258977 12520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:57.302037 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.301408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45130)
0808 15:49:57.301676 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.301803 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 15:49:57.303268 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.302819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45142)
0808 15:49:57.303046 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.303135 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260808 15:49:57.386195 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 12314
I20260808 15:49:57.415724 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 11729
I20260808 15:49:57.416977 11824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:49:57.531220 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 11729
I20260808 15:49:57.531265 12552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:49:57.530801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39557 (local address 127.0.0.1:45144)
0808 15:49:57.530997 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0808 15:49:57.531084 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
2026-08-08T15:49:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4951 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:49:57 dist-test-slave-twwt krb5kdc[12605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:49:57 dist-test-slave-twwt krb5kdc[12605](info): set up 2 sockets
Aug 08 15:49:57 dist-test-slave-twwt krb5kdc[12605](info): commencing operation
krb5kdc: starting...
W20260808 15:49:58.878355 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:49:59 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:49:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:49:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/rangeradmin_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/rangeradmin_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/rangerlookup_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/rangerlookup_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
I20260808 15:49:59.337647 27987 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzIk6P5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/postgres -l logfile start
2026-08-08 15:50:02.018 UTC [12651] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 15:50:02.018 UTC [12651] LOG: listening on IPv4 address "127.27.84.212", port 55669
2026-08-08 15:50:02.022 UTC [12651] LOG: listening on Unix socket "/tmp/.s.PGSQL.55669"
2026-08-08 15:50:02.044 UTC [12653] LOG: database system was shut down at 2026-08-08 15:50:00 UTC
2026-08-08 15:50:02.049 UTC [12651] LOG: database system is ready to accept connections
I20260808 15:50:03.267184 27987 mini_postgres.cc:96] Postgres bound to 55669
2026-08-08 15:50:03.294 UTC [12669] FATAL: database "slave" does not exist
127.27.84.212:55669 - accepting connections
I20260808 15:50:03.294993 27987 mini_ranger.cc:161] Starting Ranger...
I20260808 15:50:03.329761 27987 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 15:50:03.857795 27987 mini_ranger.cc:87] Created ranger Postgres database
I20260808 15:50:03.857976 27987 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-admin
2026-08-08 15:50:04,174 [I] DB FLAVOR :POSTGRES
2026-08-08 15:50:04,175 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:50:04,175 [I] Checking connection..
2026-08-08 15:50:04,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 15:50:04,446 [I] Checking connection passed.
2026-08-08 15:50:04,447 [I] --------- Verifying version history table ---------
2026-08-08 15:50:04,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:50:04,721 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:50:04,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:50:04,991 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:50:04,991 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 15:50:04,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 15:50:05.231 UTC [12766] WARNING: there is no transaction in progress
2026-08-08 15:50:05,241 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 15:50:05,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:50:05,514 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 15:50:05,515 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:50:05,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 15:50:05,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 15:50:06,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-twwt.c.gcp-upstream.internal','N') ;"
2026-08-08 15:50:06,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:50:06,562 [I] Table x_portal_user does not exist in database ranger
2026-08-08 15:50:06,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:50:06,833 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 15:50:06,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:50:07,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:50:07,321 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 15:50:07,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 15:50:08.301 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.318 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.335 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.352 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.368 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.384 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.440 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.447 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.462 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.472 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.483 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.498 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.508 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.515 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.522 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:08.530 UTC [12943] WARNING: there is no transaction in progress
2026-08-08 15:50:09,218 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 15:50:09,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 15:50:09,488 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 15:50:09,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:50:09,732 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 15:50:09,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:50:09,999 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 15:50:09,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:50:10,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:50:10,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-twwt.c.gcp-upstream.internal';"
2026-08-08 15:50:10,760 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 15:50:10,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 15:50:11,015 [I] DB_PATCHES have already been applied
I20260808 15:50:11.021550 27987 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzIk6P5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 15:50:11.021644 27987 mini_ranger.cc:193] Using host: 127.27.84.212
I20260808 15:50:11.046280 27987 mini_ranger.cc:227] Ranger admin URL: http://127.27.84.212:34769
Aug 08, 2026 3:50:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:50:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:50:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:50:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:50:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:50:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:50:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 15:50:12.452299 27987 mini_ranger.cc:160] Time spent starting Ranger: real 9.157s user 0.004s sys 0.070s
Aug 08, 2026 3:50:33 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:50:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 3:50:33 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:50:33 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:50:34 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 15:50:34.782364 27987 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/rangerkms_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/rangerkms_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.84.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.84.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 15:50:35 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 15:50:35.021711 27987 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 15:50:35.053424 27987 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 15:50:36.062062 27987 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 15:50:36.062259 27987 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms
I20260808 15:50:36.062407 27987 mini_ranger_kms.cc:227] Using postgres at 127.27.84.212:55669
2026-08-08 15:50:36,547 [I] DB FLAVOR :POSTGRES
2026-08-08 15:50:36,547 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:50:36,547 [I] Checking connection
2026-08-08 15:50:36,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 15:50:36,802 [I] connection success
2026-08-08 15:50:36,802 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 15:50:36,802 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 15:50:36,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 15:50:37,063 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 15:50:37,064 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:50:37,064 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 15:50:37,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:55669/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 15:50:37,356 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 15:50:37.839857 27987 mini_ranger_kms.cc:310] Created kms service
I20260808 15:50:38.058430 27987 mini_ranger_kms.cc:326] Created kudu user
I20260808 15:50:38.107016 27987 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 15:50:38 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:50:38.540670 27987 mini_ranger_kms.cc:384] Added ranger policy
I20260808 15:50:38.542162 27987 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 15:50:38.542248 27987 mini_ranger_kms.cc:240] Using host: 127.27.84.212
I20260808 15:50:38.578142 27987 mini_ranger_kms.cc:276] Ranger KMS PID: 13219
I20260808 15:50:38.578373 27987 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.84.212:40279
Aug 08, 2026 3:50:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:50:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:50:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:50:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:50:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:50:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 15:50:40 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.84.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 08 15:50:41 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.84.212@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:50:41.470867 27987 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.449s user 0.116s sys 0.423s
I20260808 15:50:41.471252 27987 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 15:50:41.471498 27987 mini_ranger_kms.cc:401] 127.27.84.212:40279/kms/v1/keys
Aug 08 15:50:41 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.84.212@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
Aug 08 15:50:43 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:50:44.080425 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:38557
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:38557
--ranger_config_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:40279/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:50:44.459177 13292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:50:44.459502 13292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:50:44.459674 13292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:50:44.467265 13292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:50:44.467379 13292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:50:44.467494 13292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:50:44.467597 13292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:50:44.467687 13292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:50:44.478327 13292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.84.212:40279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:38557
--ranger_config_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:50:44.479795 13292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:50:44.481268 13292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:50:44.494503 13301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:44.494669 13300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:44.494681 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:50:44.495312 13292 server_base.cc:1034] running on GCE node
I20260808 15:50:44.496229 13292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:50:44.497644 13292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:50:44.498893 13292 hybrid_clock.cc:648] HybridClock initialized: now 1786204244498840 us; error 43 us; skew 500 ppm
I20260808 15:50:44.499395 13292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:50:44 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204244, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:50:44.502782 13292 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:50:44.504410 13292 webserver.cc:460] Webserver started at http://127.27.84.254:37261/ using document root <none> and password file <none>
I20260808 15:50:44.505087 13292 fs_manager.cc:359] Metadata directory not provided
I20260808 15:50:44.505234 13292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:50:44.505553 13292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:50:44 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204244, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:50:44.563122 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "a14d85bb79064fb3a2f1142bd19ecc1b"
format_stamp: "Formatted at 2026-08-08 15:50:44 on dist-test-slave-twwt"
server_key: "73c88aff4bf8e8fece4336b2505b72d3"
server_key_iv: "c6691015b211aa8ffe58c765709be588"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:44.564381 13292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "a14d85bb79064fb3a2f1142bd19ecc1b"
format_stamp: "Formatted at 2026-08-08 15:50:44 on dist-test-slave-twwt"
server_key: "73c88aff4bf8e8fece4336b2505b72d3"
server_key_iv: "c6691015b211aa8ffe58c765709be588"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:44.572355 13292 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260808 15:50:44.593698 13317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:44.595698 13292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:50:44.595923 13292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "a14d85bb79064fb3a2f1142bd19ecc1b"
format_stamp: "Formatted at 2026-08-08 15:50:44 on dist-test-slave-twwt"
server_key: "73c88aff4bf8e8fece4336b2505b72d3"
server_key_iv: "c6691015b211aa8ffe58c765709be588"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:44.596203 13292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:50:44.618395 13292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:50:44.622678 13292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:50:44.622956 13292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:50:44.642577 13292 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:38557
I20260808 15:50:44.642692 13390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:38557 every 8 connection(s)
I20260808 15:50:44.644022 13292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:50:44.648098 13391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:50:44.648794 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 13292
I20260808 15:50:44.649019 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:50:44.649963 27987 external_mini_cluster.cc:1428] Setting key 59e2a0d561d2c2d4e4691c987a7158f9
I20260808 15:50:44.663266 13391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b: Bootstrap starting.
Aug 08 15:50:44 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:50:44.667447 13391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b: Neither blocks nor log segments found. Creating new log.
I20260808 15:50:44.668627 13391 log.cc:824] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:50:44.671975 13391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b: No bootstrap required, opened a new log
I20260808 15:50:44.674530 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:44.652333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.27.84.254:38557)
0808 15:50:44.653343 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:44.653356 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:44.653397 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:50:44.655258 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:44.655282 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:44.655693 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:44.656047 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:44.657618 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:44.659115 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:44.660996 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:44.661687 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:44.661841 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:44.666654 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:44.666666 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:44.666685 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:44.666733 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:44.670225 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:44.671043 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:44.671050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:44.671057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:44.671295 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:44.671700 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:44.671706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:44.671709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:44.672304 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:44.672628 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:44.673079 (+ 451us) server_negotiation.cc:299] Negotiation successful
0808 15:50:44.673497 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":478,"threads_started":1}
I20260808 15:50:44.680819 13391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38557 } }
I20260808 15:50:44.681326 13391 raft_consensus.cc:374] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:50:44.681470 13391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a14d85bb79064fb3a2f1142bd19ecc1b, State: Initialized, Role: FOLLOWER
I20260808 15:50:44.682049 13391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38557 } }
I20260808 15:50:44.682293 13391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:50:44.682432 13391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:50:44.682657 13391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:50:44.684967 13391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38557 } }
I20260808 15:50:44.685519 13391 leader_election.cc:302] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a14d85bb79064fb3a2f1142bd19ecc1b; no voters:
I20260808 15:50:44.686160 13391 leader_election.cc:288] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:50:44.686358 13405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:50:44.687254 13405 raft_consensus.cc:686] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [term 1 LEADER]: Becoming Leader. State: Replica: a14d85bb79064fb3a2f1142bd19ecc1b, State: Running, Role: LEADER
I20260808 15:50:44.688022 13405 consensus_queue.cc:237] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38557 } }
I20260808 15:50:44.688936 13391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:50:44.691610 13408 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a14d85bb79064fb3a2f1142bd19ecc1b. Latest consensus state: current_term: 1 leader_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38557 } } }
I20260808 15:50:44.691602 13407 sys_catalog.cc:455] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a14d85bb79064fb3a2f1142bd19ecc1b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38557 } } }
I20260808 15:50:44.692072 13408 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [sys.catalog]: This master's current role is: LEADER
I20260808 15:50:44.692130 13407 sys_catalog.cc:458] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b [sys.catalog]: This master's current role is: LEADER
I20260808 15:50:44.694193 13416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:50:44.700147 13416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:50:44.700970 13391 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:50:44.722735 13416 catalog_manager.cc:1347] Generated new cluster ID: f867c7c5f0c946978e027599fb59445f
I20260808 15:50:44.722925 13416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:50:44.745873 13416 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:50:44.747998 13416 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:50:44.766520 13416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b: Generated new TSK 0
I20260808 15:50:44.767764 13416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:50:45.556420 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:40279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:38557
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:50:45 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204245, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:50:46.089690 13461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:50:46.090260 13461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:50:46.090523 13461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:50:46.100739 13461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:50:46.100921 13461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:50:46.101150 13461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:50:46.113243 13461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.84.212:40279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:50:46.114954 13461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:50:46.116484 13461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:50:46.135391 13479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:46.137991 13480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:46.146238 13482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:50:46.147423 13461 server_base.cc:1034] running on GCE node
I20260808 15:50:46.148478 13461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:50:46.155009 13461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:50:46.156399 13461 hybrid_clock.cc:648] HybridClock initialized: now 1786204246156254 us; error 133 us; skew 500 ppm
I20260808 15:50:46.156951 13461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:50:46 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204246, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:50:46.162355 13461 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:50:46.168025 13461 webserver.cc:460] Webserver started at http://127.27.84.193:38409/ using document root <none> and password file <none>
I20260808 15:50:46.169165 13461 fs_manager.cc:359] Metadata directory not provided
I20260808 15:50:46.169351 13461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:50:46.169791 13461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:50:46 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204246, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:50:46.197520 13461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "23b002d9be8a4f18b0332691917804c8"
format_stamp: "Formatted at 2026-08-08 15:50:46 on dist-test-slave-twwt"
server_key: "479b5b1992c8c6aec41c8c3252f07afe"
server_key_iv: "e756e5b0520ad6dc7939c116943b07d8"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:46.198855 13461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "23b002d9be8a4f18b0332691917804c8"
format_stamp: "Formatted at 2026-08-08 15:50:46 on dist-test-slave-twwt"
server_key: "479b5b1992c8c6aec41c8c3252f07afe"
server_key_iv: "e756e5b0520ad6dc7939c116943b07d8"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:46.207584 13461 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20260808 15:50:46.228237 13497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:46.230630 13461 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:50:46.230850 13461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "23b002d9be8a4f18b0332691917804c8"
format_stamp: "Formatted at 2026-08-08 15:50:46 on dist-test-slave-twwt"
server_key: "479b5b1992c8c6aec41c8c3252f07afe"
server_key_iv: "e756e5b0520ad6dc7939c116943b07d8"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:46.231140 13461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:50:46.275075 13461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:50:46.281780 13461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:50:46.282311 13461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:50:46.287127 13461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:50:46.289624 13461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:50:46.289790 13461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:46.290057 13461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:50:46.290145 13461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:46.375797 13461 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:42867
I20260808 15:50:46.375821 13676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:42867 every 8 connection(s)
I20260808 15:50:46.377533 13461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:50:46.380759 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 13461
I20260808 15:50:46.381031 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:50:46.382084 27987 external_mini_cluster.cc:1428] Setting key 6db17133b8e2ec84ee36a61878da50d4
Aug 08 15:50:46 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204246, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:50:46.424938 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:46.384063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:38557 (local address 127.27.84.193:54605)
0808 15:50:46.385145 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:46.385270 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:46.389151 (+ 3881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:46.402274 (+ 13123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:46.402286 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:46.403079 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:50:46.403834 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:46.403858 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:46.405689 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:46.405698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:46.407236 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:46.407250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:46.409017 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:46.409026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:46.409382 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:46.410268 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:50:46.410304 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:50:46.413777 (+ 3473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:50:46.418998 (+ 5221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:50:46.419010 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:50:46.419033 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:50:46.419809 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:50:46.420365 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:50:46.420373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:50:46.420379 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:50:46.420650 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:50:46.421534 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:50:46.421545 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:50:46.422118 (+ 573us) client_negotiation.cc:769] Sending connection context
0808 15:50:46.422452 (+ 334us) client_negotiation.cc:240] Negotiation successful
0808 15:50:46.423170 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":5504,"threads_started":1}
I20260808 15:50:46.426429 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:46.383019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:54605 (local address 127.27.84.254:38557)
0808 15:50:46.401266 (+ 18247us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:46.401275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:46.401306 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:50:46.401385 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:46.401393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:46.401507 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:46.402761 (+ 1254us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:46.404034 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:46.405428 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:46.407393 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:46.408447 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:46.408592 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:46.414091 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:46.414101 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:46.414109 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:46.414158 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:46.418805 (+ 4647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:46.419989 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:46.419997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:46.420003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:46.420217 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:46.420830 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:46.420838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:46.420844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:46.421288 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:46.421452 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:46.425617 (+ 4165us) server_negotiation.cc:299] Negotiation successful
0808 15:50:46.425925 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18067,"thread_start_us":18575,"threads_started":1}
I20260808 15:50:46.429338 13677 heartbeater.cc:347] Connected to a master server at 127.27.84.254:38557
I20260808 15:50:46.429684 13677 heartbeater.cc:464] Registering TS with master...
I20260808 15:50:46.430756 13677 heartbeater.cc:511] Master 127.27.84.254:38557 requested a full tablet report, sending...
I20260808 15:50:46.433638 13340 ts_manager.cc:194] Registered new tserver with Master: 23b002d9be8a4f18b0332691917804c8 (127.27.84.193:42867)
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
I20260808 15:50:46.435999 13340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:54605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Aug 08 15:50:46 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204245, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
W20260808 15:50:46.488050 27987 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.84.194: real 0.101s user 0.009s sys 0.021s
I20260808 15:50:46.489173 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:40279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:38557
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:50:46 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204245, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
W20260808 15:50:47.091437 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:50:47.091907 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:50:47.092099 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:50:47.102396 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:50:47.102543 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:50:47.102741 13689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:50:47.114012 13689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.84.212:40279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:50:47.115368 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:50:47.116878 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:50:47.132875 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:47.132942 13710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:47.132871 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:50:47.134196 13689 server_base.cc:1034] running on GCE node
I20260808 15:50:47.134909 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:50:47.135951 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:50:47.137228 13689 hybrid_clock.cc:648] HybridClock initialized: now 1786204247137139 us; error 43 us; skew 500 ppm
I20260808 15:50:47.137637 13689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:50:47 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:50:47.141397 13689 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:50:47.143383 13689 webserver.cc:460] Webserver started at http://127.27.84.194:45967/ using document root <none> and password file <none>
I20260808 15:50:47.144198 13689 fs_manager.cc:359] Metadata directory not provided
I20260808 15:50:47.144356 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:50:47.144776 13689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:50:47 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:50:47.160419 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "e08e85491d28410b93215df35f7b7ece"
format_stamp: "Formatted at 2026-08-08 15:50:47 on dist-test-slave-twwt"
server_key: "e9755aabd92a4b94f912c6c3939e7346"
server_key_iv: "f57e6c893d4dfb4521b55c5e31e8543c"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:47.161502 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "e08e85491d28410b93215df35f7b7ece"
format_stamp: "Formatted at 2026-08-08 15:50:47 on dist-test-slave-twwt"
server_key: "e9755aabd92a4b94f912c6c3939e7346"
server_key_iv: "f57e6c893d4dfb4521b55c5e31e8543c"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:47.168053 13689 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260808 15:50:47.186760 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:47.188508 13689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 15:50:47.188740 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "e08e85491d28410b93215df35f7b7ece"
format_stamp: "Formatted at 2026-08-08 15:50:47 on dist-test-slave-twwt"
server_key: "e9755aabd92a4b94f912c6c3939e7346"
server_key_iv: "f57e6c893d4dfb4521b55c5e31e8543c"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:47.189024 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:50:47.207876 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:50:47.212322 13689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:50:47.212639 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:50:47.213860 13689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:50:47.215567 13689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:50:47.215651 13689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:47.215740 13689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:50:47.215778 13689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:47.266485 13689 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:43921
I20260808 15:50:47.266513 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:43921 every 8 connection(s)
I20260808 15:50:47.268347 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:50:47.275441 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 13689
I20260808 15:50:47.275648 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:50:47.276198 27987 external_mini_cluster.cc:1428] Setting key c35f7081f30061bed338ece9b9b4596c
Aug 08 15:50:47 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:50:47.293495 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:47.271669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:49243 (local address 127.27.84.254:38557)
0808 15:50:47.272299 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:47.272303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:47.273266 (+ 963us) server_negotiation.cc:407] Connection header received
0808 15:50:47.275388 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:47.275396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:47.275583 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:47.275719 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:47.277639 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:47.278435 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:47.280139 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:47.280969 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:47.281116 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:47.285068 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:47.285077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:47.285085 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:47.285131 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:47.289362 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:47.290361 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:47.290369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:47.290375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:47.290576 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:47.291064 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:47.291071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:47.291077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:47.291478 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:47.291639 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:47.292815 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0808 15:50:47.293048 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":383,"threads_started":1}
I20260808 15:50:47.294981 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:47.272235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:38557 (local address 127.27.84.194:49243)
0808 15:50:47.273075 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:47.273115 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:47.275101 (+ 1986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:47.276034 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:47.276043 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:47.276621 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:50:47.277464 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:47.277486 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:47.278686 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:47.278690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:47.279976 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:47.279990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:47.281192 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:47.281197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:47.281494 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:47.282368 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:50:47.282398 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:50:47.284808 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:50:47.289515 (+ 4707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:50:47.289522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:50:47.289536 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:50:47.290194 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:50:47.290714 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:50:47.290718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:50:47.290720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:50:47.290938 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:50:47.291668 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:50:47.291679 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:50:47.292210 (+ 531us) client_negotiation.cc:769] Sending connection context
0808 15:50:47.292606 (+ 396us) client_negotiation.cc:240] Negotiation successful
0808 15:50:47.293276 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":389,"threads_started":1}
I20260808 15:50:47.296844 13899 heartbeater.cc:347] Connected to a master server at 127.27.84.254:38557
I20260808 15:50:47.297241 13899 heartbeater.cc:464] Registering TS with master...
I20260808 15:50:47.298199 13899 heartbeater.cc:511] Master 127.27.84.254:38557 requested a full tablet report, sending...
I20260808 15:50:47.299974 13340 ts_manager.cc:194] Registered new tserver with Master: e08e85491d28410b93215df35f7b7ece (127.27.84.194:43921)
I20260808 15:50:47.301312 13340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:49243
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:50:47.351087 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:40279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:38557
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:50:47.444932 13677 heartbeater.cc:503] Master 127.27.84.254:38557 was elected leader, sending a full tablet report...
W20260808 15:50:47.804714 13909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:50:47.805148 13909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:50:47.805416 13909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:50:47.813844 13909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:50:47.814000 13909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:50:47.814178 13909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:50:47.825399 13909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.84.212:40279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:38557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:50:47.826903 13909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:50:47.828286 13909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:50:47.844689 13919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:47.844501 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:50:47.844759 13920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:50:47.845882 13909 server_base.cc:1034] running on GCE node
I20260808 15:50:47.846529 13909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:50:47.847556 13909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:50:47.848829 13909 hybrid_clock.cc:648] HybridClock initialized: now 1786204247848753 us; error 56 us; skew 500 ppm
I20260808 15:50:47.849174 13909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:50:47 dist-test-slave-twwt krb5kdc[12605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:50:47.853001 13909 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:50:47.854912 13909 webserver.cc:460] Webserver started at http://127.27.84.195:40325/ using document root <none> and password file <none>
I20260808 15:50:47.855701 13909 fs_manager.cc:359] Metadata directory not provided
I20260808 15:50:47.855861 13909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:50:47.856218 13909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:50:47 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:50:47.873334 13909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "bac5ac8b2d9d41efb07d4cade72f64a7"
format_stamp: "Formatted at 2026-08-08 15:50:47 on dist-test-slave-twwt"
server_key: "d20ac71b2818025f550ee89666e315b8"
server_key_iv: "8620de6d48a1f7b3e66ea303c5b647fa"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:47.874260 13909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "bac5ac8b2d9d41efb07d4cade72f64a7"
format_stamp: "Formatted at 2026-08-08 15:50:47 on dist-test-slave-twwt"
server_key: "d20ac71b2818025f550ee89666e315b8"
server_key_iv: "8620de6d48a1f7b3e66ea303c5b647fa"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:47.881126 13909 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20260808 15:50:47.898533 13934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:47.900315 13909 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260808 15:50:47.900604 13909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "bac5ac8b2d9d41efb07d4cade72f64a7"
format_stamp: "Formatted at 2026-08-08 15:50:47 on dist-test-slave-twwt"
server_key: "d20ac71b2818025f550ee89666e315b8"
server_key_iv: "8620de6d48a1f7b3e66ea303c5b647fa"
server_key_version: "kuduclusterkey@0"
I20260808 15:50:47.900885 13909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:50:47.925758 13909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:50:47.930773 13909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:50:47.931171 13909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:50:47.932693 13909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:50:47.934696 13909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:50:47.934876 13909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260808 15:50:47.935072 13909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:50:47.935169 13909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:50:48.005491 13909 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:46033
I20260808 15:50:48.005492 14112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:46033 every 8 connection(s)
I20260808 15:50:48.007143 13909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:50:48.015885 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 13909
I20260808 15:50:48.016120 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:50:48.016630 27987 external_mini_cluster.cc:1428] Setting key f820ed31023228757f24c2bc4cc93f92
Aug 08 15:50:48 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204247, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:50:48.032604 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.010257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:60907 (local address 127.27.84.254:38557)
0808 15:50:48.011090 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.011095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.012335 (+ 1240us) server_negotiation.cc:407] Connection header received
0808 15:50:48.015084 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.015092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.015275 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.015424 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:48.017300 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.018397 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.020902 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.021430 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.021572 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.025420 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:48.025425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:48.025429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:48.025467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:48.028661 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.029462 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.029466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.029469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.029666 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.030216 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.030220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.030222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.030545 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:48.030749 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.031766 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.032047 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":571,"threads_started":1}
I20260808 15:50:48.033800 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.010874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:38557 (local address 127.27.84.195:60907)
0808 15:50:48.012089 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:48.012129 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:48.014769 (+ 2640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:48.015737 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:48.015747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:48.016394 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:50:48.017102 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.017123 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.018658 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.018664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.020739 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.020754 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.021798 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.021807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.022008 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.022817 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:50:48.022849 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:50:48.025153 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:50:48.028827 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:50:48.028848 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:50:48.028861 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:50:48.029321 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:50:48.029792 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:50:48.029798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:50:48.029802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:50:48.030080 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:50:48.030750 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:50:48.030756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:50:48.031262 (+ 506us) client_negotiation.cc:769] Sending connection context
0808 15:50:48.031663 (+ 401us) client_negotiation.cc:240] Negotiation successful
0808 15:50:48.032307 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":540,"threads_started":1}
I20260808 15:50:48.035444 14113 heartbeater.cc:347] Connected to a master server at 127.27.84.254:38557
I20260808 15:50:48.035853 14113 heartbeater.cc:464] Registering TS with master...
I20260808 15:50:48.036921 14113 heartbeater.cc:511] Master 127.27.84.254:38557 requested a full tablet report, sending...
I20260808 15:50:48.038689 13340 ts_manager.cc:194] Registered new tserver with Master: bac5ac8b2d9d41efb07d4cade72f64a7 (127.27.84.195:46033)
I20260808 15:50:48.039574 13340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:60907
I20260808 15:50:48.045158 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:50:48.065105 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.050358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49384 (local address 127.27.84.254:38557)
0808 15:50:48.050566 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.050575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.051142 (+ 567us) server_negotiation.cc:407] Connection header received
0808 15:50:48.051284 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.051289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.051364 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.051446 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:48.053064 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.053792 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.055381 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.055905 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.055999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.058907 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:48.058912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:48.058915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:48.058947 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:48.062023 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.062761 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.062766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.062768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.062928 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.063532 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.063536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.063539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.063977 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:48.064194 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.064524 (+ 330us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.064726 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:50:48.069679 13340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:50:48.078326 13340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:50:48.115320 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.092639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:46033 (local address 127.0.0.1:33154)
0808 15:50:48.093836 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:48.093891 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:48.094058 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:48.104675 (+ 10617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:48.104682 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:48.104713 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:50:48.105718 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.105740 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.108567 (+ 2827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.108572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.111637 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.111650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.114074 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.114081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.114363 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.114426 (+ 63us) client_negotiation.cc:769] Sending connection context
0808 15:50:48.114540 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:50:48.114783 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":8,"thread_start_us":646,"threads_started":1}
I20260808 15:50:48.115531 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.092916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.27.84.195:46033)
0808 15:50:48.103798 (+ 10882us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.103812 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.103877 (+ 65us) server_negotiation.cc:407] Connection header received
0808 15:50:48.104012 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.104024 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.104397 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.104893 (+ 496us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:50:48.106035 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.108383 (+ 2348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.111964 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.113813 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.113947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.114270 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.114700 (+ 430us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.114925 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10554,"thread_start_us":10319,"threads_started":1}
I20260808 15:50:48.115953 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.093413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.27.84.193:42867)
0808 15:50:48.104336 (+ 10923us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.104346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.104381 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:50:48.104489 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.104495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.104881 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.105095 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:50:48.106389 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.107685 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.111766 (+ 4081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.113813 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.113947 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.114256 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.115260 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.115447 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10668,"thread_start_us":10467,"threads_started":1}
I20260808 15:50:48.116954 14137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.092822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42867 (local address 127.0.0.1:54676)
0808 15:50:48.102591 (+ 9769us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:48.102651 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:48.102777 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:48.105081 (+ 2304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:48.105086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:48.105119 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:50:48.106128 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.106142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.107833 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.107841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.111589 (+ 3748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.111605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.114790 (+ 3185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.114796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.115020 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.115065 (+ 45us) client_negotiation.cc:769] Sending connection context
0808 15:50:48.116229 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0808 15:50:48.116340 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9623,"mutex_wait_us":76,"thread_start_us":9482,"threads_started":1}
I20260808 15:50:48.118069 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.102972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.27.84.194:43921)
0808 15:50:48.104677 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.104685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.109153 (+ 4468us) server_negotiation.cc:407] Connection header received
0808 15:50:48.109413 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.109421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.109720 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.109954 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:50:48.110966 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.112361 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.114424 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.116459 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.116850 (+ 391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.117053 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.117634 (+ 581us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.117835 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1512,"thread_start_us":1405,"threads_started":1}
I20260808 15:50:48.118222 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.104683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:43921 (local address 127.0.0.1:39224)
0808 15:50:48.109000 (+ 4317us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:48.109024 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:48.109154 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:48.109969 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:48.109976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:48.110004 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:50:48.110737 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.110775 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.112515 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.112519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.114258 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.114283 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.117234 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.117241 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.117441 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.117473 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:50:48.117586 (+ 113us) client_negotiation.cc:240] Negotiation successful
0808 15:50:48.117726 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4118,"mutex_wait_us":5,"thread_start_us":3750,"threads_started":1}
I20260808 15:50:48.120487 14010 tablet_service.cc:1467] Processing CreateTablet for tablet 82b87731662c488bb0de42d99c0e4d0c (DEFAULT_TABLE table=test-table [id=6250b3c0917b4dab90136f1d36451746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:50:48.121765 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b87731662c488bb0de42d99c0e4d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:50:48.122464 13566 tablet_service.cc:1467] Processing CreateTablet for tablet 82b87731662c488bb0de42d99c0e4d0c (DEFAULT_TABLE table=test-table [id=6250b3c0917b4dab90136f1d36451746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:50:48.122522 13806 tablet_service.cc:1467] Processing CreateTablet for tablet 82b87731662c488bb0de42d99c0e4d0c (DEFAULT_TABLE table=test-table [id=6250b3c0917b4dab90136f1d36451746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:50:48.124131 13566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b87731662c488bb0de42d99c0e4d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:50:48.124105 13806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b87731662c488bb0de42d99c0e4d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:50:48.138022 14145 tablet_bootstrap.cc:492] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Bootstrap starting.
I20260808 15:50:48.141904 14145 tablet_bootstrap.cc:654] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Neither blocks nor log segments found. Creating new log.
I20260808 15:50:48.142393 14146 tablet_bootstrap.cc:492] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Bootstrap starting.
I20260808 15:50:48.144229 14147 tablet_bootstrap.cc:492] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Bootstrap starting.
I20260808 15:50:48.144537 14145 log.cc:824] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:50:48.148588 14147 tablet_bootstrap.cc:654] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Neither blocks nor log segments found. Creating new log.
I20260808 15:50:48.148975 14146 tablet_bootstrap.cc:654] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Neither blocks nor log segments found. Creating new log.
I20260808 15:50:48.149833 14147 log.cc:824] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Log is configured to *not* fsync() on all Append() calls
I20260808 15:50:48.150753 14146 log.cc:824] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:50:48.153371 14147 tablet_bootstrap.cc:492] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: No bootstrap required, opened a new log
I20260808 15:50:48.153713 14147 ts_tablet_manager.cc:1397] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260808 15:50:48.154075 14145 tablet_bootstrap.cc:492] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: No bootstrap required, opened a new log
I20260808 15:50:48.154445 14145 ts_tablet_manager.cc:1397] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260808 15:50:48.155050 14146 tablet_bootstrap.cc:492] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: No bootstrap required, opened a new log
I20260808 15:50:48.155462 14146 ts_tablet_manager.cc:1397] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260808 15:50:48.163255 14147 raft_consensus.cc:348] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.163739 14147 raft_consensus.cc:374] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:50:48.163890 14147 raft_consensus.cc:729] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08e85491d28410b93215df35f7b7ece, State: Initialized, Role: FOLLOWER
I20260808 15:50:48.164669 14147 consensus_queue.cc:260] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.166065 13899 heartbeater.cc:503] Master 127.27.84.254:38557 was elected leader, sending a full tablet report...
I20260808 15:50:48.166903 14147 ts_tablet_manager.cc:1428] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Time spent starting tablet: real 0.013s user 0.015s sys 0.001s
I20260808 15:50:48.169080 14145 raft_consensus.cc:348] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.169687 14145 raft_consensus.cc:374] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:50:48.169845 14145 raft_consensus.cc:729] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac5ac8b2d9d41efb07d4cade72f64a7, State: Initialized, Role: FOLLOWER
I20260808 15:50:48.170234 14146 raft_consensus.cc:348] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.170773 14146 raft_consensus.cc:374] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:50:48.170902 14146 raft_consensus.cc:729] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b002d9be8a4f18b0332691917804c8, State: Initialized, Role: FOLLOWER
I20260808 15:50:48.170688 14145 consensus_queue.cc:260] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.171651 14146 consensus_queue.cc:260] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.174314 14146 ts_tablet_manager.cc:1428] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Time spent starting tablet: real 0.019s user 0.016s sys 0.001s
I20260808 15:50:48.175547 14113 heartbeater.cc:503] Master 127.27.84.254:38557 was elected leader, sending a full tablet report...
I20260808 15:50:48.176857 14145 ts_tablet_manager.cc:1428] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Time spent starting tablet: real 0.022s user 0.019s sys 0.004s
W20260808 15:50:48.259498 14114 tablet.cc:2369] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:50:48.270910 13900 tablet.cc:2369] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:50:48.291487 14152 raft_consensus.cc:482] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:50:48.291859 14152 raft_consensus.cc:504] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.293975 14152 leader_election.cc:288] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bac5ac8b2d9d41efb07d4cade72f64a7 (127.27.84.195:46033), 23b002d9be8a4f18b0332691917804c8 (127.27.84.193:42867)
I20260808 15:50:48.303184 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.294890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:45885 (local address 127.27.84.195:46033)
0808 15:50:48.295323 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.295328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.296461 (+ 1133us) server_negotiation.cc:407] Connection header received
0808 15:50:48.296693 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.296699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.296805 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.296911 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:50:48.297824 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.298692 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.300874 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.301942 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.302162 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.302246 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.302688 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.302819 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260808 15:50:48.303201 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.294957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:42837 (local address 127.27.84.193:42867)
0808 15:50:48.295256 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.295265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.296494 (+ 1229us) server_negotiation.cc:407] Connection header received
0808 15:50:48.296805 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.296809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.296906 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.296998 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:50:48.298186 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.299093 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.300931 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.302063 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.302291 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.302371 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.302770 (+ 399us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.302902 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 15:50:48.303483 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.294278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:42867 (local address 127.27.84.194:42837)
0808 15:50:48.296347 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:48.296363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:48.296639 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:48.297255 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:48.297260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:48.297283 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:50:48.298023 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.298034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.299240 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.299248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.300734 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.300747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.302318 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.302326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.302546 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.302571 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:50:48.302716 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 15:50:48.302851 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1875,"mutex_wait_us":40,"spinlock_wait_cycles":24832,"thread_start_us":1437,"threads_started":1}
I20260808 15:50:48.303624 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.294927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:46033 (local address 127.27.84.194:45885)
0808 15:50:48.296340 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0808 15:50:48.296357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:50:48.296524 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:50:48.296925 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:50:48.296932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:50:48.296964 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:50:48.297594 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.297615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.298833 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.298840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.300704 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:50:48.300717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.302163 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.302170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:50:48.302462 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.302523 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 15:50:48.302663 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 15:50:48.302806 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1296,"mutex_wait_us":13,"thread_start_us":1016,"threads_started":1}
I20260808 15:50:48.304699 13605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b87731662c488bb0de42d99c0e4d0c" candidate_uuid: "e08e85491d28410b93215df35f7b7ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b002d9be8a4f18b0332691917804c8" is_pre_election: true
I20260808 15:50:48.304898 14039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b87731662c488bb0de42d99c0e4d0c" candidate_uuid: "e08e85491d28410b93215df35f7b7ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" is_pre_election: true
I20260808 15:50:48.305454 13605 raft_consensus.cc:2454] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08e85491d28410b93215df35f7b7ece in term 0.
I20260808 15:50:48.305481 14039 raft_consensus.cc:2454] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e08e85491d28410b93215df35f7b7ece in term 0.
I20260808 15:50:48.306375 13722 leader_election.cc:302] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23b002d9be8a4f18b0332691917804c8, bac5ac8b2d9d41efb07d4cade72f64a7, e08e85491d28410b93215df35f7b7ece; no voters:
I20260808 15:50:48.306910 14152 raft_consensus.cc:2781] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:50:48.307171 14152 raft_consensus.cc:482] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:50:48.307319 14152 raft_consensus.cc:3037] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:50:48.309907 14152 raft_consensus.cc:504] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.310531 14152 leader_election.cc:288] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [CANDIDATE]: Term 1 election: Requested vote from peers bac5ac8b2d9d41efb07d4cade72f64a7 (127.27.84.195:46033), 23b002d9be8a4f18b0332691917804c8 (127.27.84.193:42867)
I20260808 15:50:48.311209 14039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b87731662c488bb0de42d99c0e4d0c" candidate_uuid: "e08e85491d28410b93215df35f7b7ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7"
I20260808 15:50:48.311293 13605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b87731662c488bb0de42d99c0e4d0c" candidate_uuid: "e08e85491d28410b93215df35f7b7ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b002d9be8a4f18b0332691917804c8"
I20260808 15:50:48.311499 14039 raft_consensus.cc:3037] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:50:48.311522 13605 raft_consensus.cc:3037] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:50:48.315486 14039 raft_consensus.cc:2454] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08e85491d28410b93215df35f7b7ece in term 1.
I20260808 15:50:48.315486 13605 raft_consensus.cc:2454] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e08e85491d28410b93215df35f7b7ece in term 1.
I20260808 15:50:48.315977 13722 leader_election.cc:302] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b002d9be8a4f18b0332691917804c8, e08e85491d28410b93215df35f7b7ece; no voters:
I20260808 15:50:48.316360 14152 raft_consensus.cc:2781] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:50:48.316854 14152 raft_consensus.cc:686] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 1 LEADER]: Becoming Leader. State: Replica: e08e85491d28410b93215df35f7b7ece, State: Running, Role: LEADER
I20260808 15:50:48.317381 14152 consensus_queue.cc:237] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } }
I20260808 15:50:48.322547 13340 catalog_manager.cc:5495] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece reported cstate change: term changed from 0 to 1, leader changed from <none> to e08e85491d28410b93215df35f7b7ece (127.27.84.194). New cstate: current_term: 1 leader_uuid: "e08e85491d28410b93215df35f7b7ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e08e85491d28410b93215df35f7b7ece" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 43921 } health_report { overall_health: HEALTHY } } }
I20260808 15:50:48.346968 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.340714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.27.84.194:43921)
0808 15:50:48.340920 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.340925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.340942 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:50:48.341049 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.341052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.341137 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.341266 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:50:48.341760 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.342466 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.344166 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.344648 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.344748 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.345438 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:50:48.346100 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:50:48.346246 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.346399 (+ 153us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.346489 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260808 15:50:48.356923 13605 raft_consensus.cc:1261] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 1 FOLLOWER]: Refusing update from remote peer e08e85491d28410b93215df35f7b7ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:50:48.356922 14039 raft_consensus.cc:1261] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 1 FOLLOWER]: Refusing update from remote peer e08e85491d28410b93215df35f7b7ece: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:50:48.358234 14152 consensus_queue.cc:1035] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "bac5ac8b2d9d41efb07d4cade72f64a7" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 46033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:50:48.358670 14160 consensus_queue.cc:1035] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b002d9be8a4f18b0332691917804c8" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 42867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:50:48.375443 14166 mvcc.cc:205] Tried to move back new op lower bound from 7316292601257091072 to 7316292601112023040. Current Snapshot: MvccSnapshot[applied={T|T < 7316292601257091072}]
I20260808 15:50:48.375855 14168 mvcc.cc:205] Tried to move back new op lower bound from 7316292601257091072 to 7316292601112023040. Current Snapshot: MvccSnapshot[applied={T|T < 7316292601257091072}]
I20260808 15:50:48.379012 14167 mvcc.cc:205] Tried to move back new op lower bound from 7316292601257091072 to 7316292601112023040. Current Snapshot: MvccSnapshot[applied={T|T < 7316292601257091072}]
I20260808 15:50:48.409930 13340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:50:48.410259 13340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:50:48.414671 13340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a14d85bb79064fb3a2f1142bd19ecc1b: Sending DeleteTablet for 3 replicas of tablet 82b87731662c488bb0de42d99c0e4d0c
I20260808 15:50:48.416077 14010 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b87731662c488bb0de42d99c0e4d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:50:48 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:33154
I20260808 15:50:48.416308 13566 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b87731662c488bb0de42d99c0e4d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:50:48 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:54676
I20260808 15:50:48.417683 14181 tablet_replica.cc:330] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: stopping tablet replica
I20260808 15:50:48.417775 14179 tablet_replica.cc:330] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: stopping tablet replica
I20260808 15:50:48.418288 14179 raft_consensus.cc:2229] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:50:48.418371 14181 raft_consensus.cc:2229] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:50:48.418946 14179 raft_consensus.cc:2258] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:50:48.419039 14181 raft_consensus.cc:2258] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:50:48.419848 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 13461
I20260808 15:50:48.420359 13806 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b87731662c488bb0de42d99c0e4d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:50:48 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:39224
I20260808 15:50:48.421768 14179 ts_tablet_manager.cc:1905] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:50:48.428654 14181 ts_tablet_manager.cc:1905] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:50:48.429111 14179 ts_tablet_manager.cc:1918] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:50:48.429335 14179 log.cc:1196] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/82b87731662c488bb0de42d99c0e4d0c
I20260808 15:50:48.429893 14179 ts_tablet_manager.cc:1939] T 82b87731662c488bb0de42d99c0e4d0c P bac5ac8b2d9d41efb07d4cade72f64a7: Deleting consensus metadata
I20260808 15:50:48.431926 13321 catalog_manager.cc:4841] TS bac5ac8b2d9d41efb07d4cade72f64a7 (127.27.84.195:46033): tablet 82b87731662c488bb0de42d99c0e4d0c (table test-table [id=6250b3c0917b4dab90136f1d36451746]) successfully deleted
I20260808 15:50:48.434754 14182 tablet_replica.cc:330] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: stopping tablet replica
I20260808 15:50:48.435422 14182 raft_consensus.cc:2229] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:50:48.436645 14182 raft_consensus.cc:2258] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:50:48.439826 14182 ts_tablet_manager.cc:1905] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:50:48.441681 14181 ts_tablet_manager.cc:1918] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:50:48.441985 14181 log.cc:1196] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/82b87731662c488bb0de42d99c0e4d0c
I20260808 15:50:48.442785 14181 ts_tablet_manager.cc:1939] T 82b87731662c488bb0de42d99c0e4d0c P 23b002d9be8a4f18b0332691917804c8: Deleting consensus metadata
Aug 08 15:50:48 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:50:48.445497 13318 catalog_manager.cc:4841] TS 23b002d9be8a4f18b0332691917804c8 (127.27.84.193:42867): tablet 82b87731662c488bb0de42d99c0e4d0c (table test-table [id=6250b3c0917b4dab90136f1d36451746]) successfully deleted
I20260808 15:50:48.449870 14182 ts_tablet_manager.cc:1918] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:50:48.450137 14182 log.cc:1196] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/82b87731662c488bb0de42d99c0e4d0c
I20260808 15:50:48.451378 14182 ts_tablet_manager.cc:1939] T 82b87731662c488bb0de42d99c0e4d0c P e08e85491d28410b93215df35f7b7ece: Deleting consensus metadata
I20260808 15:50:48.453470 13318 catalog_manager.cc:4841] TS e08e85491d28410b93215df35f7b7ece (127.27.84.194:43921): tablet 82b87731662c488bb0de42d99c0e4d0c (table test-table [id=6250b3c0917b4dab90136f1d36451746]) successfully deleted
I20260808 15:50:48.454680 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.421231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.27.84.193:42867)
0808 15:50:48.421476 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.421486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.430839 (+ 9353us) server_negotiation.cc:407] Connection header received
0808 15:50:48.431362 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.431374 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.431540 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.431633 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:48.434056 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.435277 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.437681 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.438422 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.438535 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.444853 (+ 6318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:48.444874 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:48.444899 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:48.444958 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:48.449679 (+ 4721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.451226 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.451238 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.451243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.451473 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.452154 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.452160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.452164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.452784 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:48.453210 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.454144 (+ 934us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.454346 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:50:48.455374 13671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:50:48.589558 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 13461
I20260808 15:50:48.625236 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 13689
Aug 08 15:50:48 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:50:48.644621 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.626108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39246 (local address 127.27.84.194:43921)
0808 15:50:48.626334 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.626340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.626365 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:50:48.626528 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.626534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.626655 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.626768 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:48.628058 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.629341 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.631002 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.631691 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.631834 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.636863 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:48.636885 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:48.636908 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:48.636963 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:48.641100 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.641879 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.641890 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.641894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.642097 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.642649 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.642654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.642658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.643298 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:48.643635 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.643963 (+ 328us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.644211 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:50:48.645676 13889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:50:48.762167 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 13689
I20260808 15:50:48.791424 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 13909
Aug 08 15:50:48 dist-test-slave-twwt krb5kdc[12605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:50:48.813073 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:50:48.792305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33162 (local address 127.27.84.195:46033)
0808 15:50:48.792516 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0808 15:50:48.792521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:50:48.792544 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:50:48.792713 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:50:48.792720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:50:48.792823 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:50:48.792964 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:50:48.794533 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.795444 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.796806 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:50:48.797177 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:50:48.797278 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:50:48.807063 (+ 9785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:50:48.807079 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:50:48.807097 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:50:48.807158 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:50:48.810235 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.811029 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.811037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.811040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.811193 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:50:48.811546 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:50:48.811550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:50:48.811552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:50:48.812008 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:50:48.812237 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:50:48.812553 (+ 316us) server_negotiation.cc:299] Negotiation successful
0808 15:50:48.812795 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:50:48.813951 14105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:50:48.907367 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 13909
I20260808 15:50:48.929368 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 13292
I20260808 15:50:48.930600 13386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:50:49.044900 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 13292
I20260808 15:50:49.078572 27987 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 15:50:49.425814 27987 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 15:50:49.426155 27987 mini_ranger.cc:66] Stopping Ranger...
I20260808 15:50:49.772105 27987 mini_ranger.cc:68] Stopped Ranger
2026-08-08 15:50:49.772 UTC [12651] LOG: received smart shutdown request
2026-08-08 15:50:49.776 UTC [12651] LOG: background worker "logical replication launcher" (PID 12663) exited with exit code 1
2026-08-08 15:50:49.777 UTC [12655] LOG: shutting down
2026-08-08 15:50:49.946 UTC [12651] LOG: database system is shut down
2026-08-08T15:50:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52563 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:50:50 dist-test-slave-twwt krb5kdc[14198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:50:50 dist-test-slave-twwt krb5kdc[14198](info): set up 2 sockets
Aug 08 15:50:50 dist-test-slave-twwt krb5kdc[14198](info): commencing operation
krb5kdc: starting...
W20260808 15:50:51.444921 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:50:51 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:50:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/rangeradmin_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/rangeradmin_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/rangerlookup_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/rangerlookup_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
I20260808 15:50:51.867848 27987 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskzIk6P5/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/postgres -l logfile start
2026-08-08 15:50:54.551 UTC [14239] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-08 15:50:54.551 UTC [14239] LOG: listening on IPv4 address "127.27.84.212", port 43963
2026-08-08 15:50:54.555 UTC [14239] LOG: listening on Unix socket "/tmp/.s.PGSQL.43963"
2026-08-08 15:50:54.580 UTC [14242] LOG: database system was shut down at 2026-08-08 15:50:53 UTC
2026-08-08 15:50:54.584 UTC [14239] LOG: database system is ready to accept connections
I20260808 15:50:55.772205 27987 mini_postgres.cc:96] Postgres bound to 43963
2026-08-08 15:50:55.798 UTC [14257] FATAL: database "slave" does not exist
127.27.84.212:43963 - accepting connections
I20260808 15:50:55.798842 27987 mini_ranger.cc:161] Starting Ranger...
I20260808 15:50:55.833248 27987 mini_ranger.cc:84] Created miniranger Postgres user
I20260808 15:50:56.341302 27987 mini_ranger.cc:87] Created ranger Postgres database
I20260808 15:50:56.341429 27987 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-admin
2026-08-08 15:50:56,635 [I] DB FLAVOR :POSTGRES
2026-08-08 15:50:56,636 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:50:56,636 [I] Checking connection..
2026-08-08 15:50:56,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-08 15:50:56,910 [I] Checking connection passed.
2026-08-08 15:50:56,911 [I] --------- Verifying version history table ---------
2026-08-08 15:50:56,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:50:57,169 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:50:57,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:50:57,440 [I] Table x_db_version_h does not exist in database ranger
2026-08-08 15:50:57,441 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-08 15:50:57,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-08 15:50:57.704 UTC [14351] WARNING: there is no transaction in progress
2026-08-08 15:50:57,714 [I] create_dbversion_catalog.sql file imported successfully
2026-08-08 15:50:57,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-08 15:50:57,976 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-08 15:50:57,977 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:50:57,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-08 15:50:58,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-08 15:50:58,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-twwt.c.gcp-upstream.internal','N') ;"
2026-08-08 15:50:58,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:50:59,003 [I] Table x_portal_user does not exist in database ranger
2026-08-08 15:50:59,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:50:59,279 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-08 15:50:59,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:50:59,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:50:59,774 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-08 15:50:59,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-08 15:51:00.795 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.813 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.830 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.846 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.864 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.883 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.950 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.957 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.972 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.982 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:00.993 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:01.008 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:01.018 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:01.025 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:01.033 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:01.041 UTC [14528] WARNING: there is no transaction in progress
2026-08-08 15:51:01,733 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-08 15:51:01,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-08 15:51:01,991 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-08 15:51:01,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-08 15:51:02,242 [I] Table x_portal_user already exists in database 'ranger'
2026-08-08 15:51:02,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-08 15:51:02,488 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-08 15:51:02,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-08 15:51:02,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-08 15:51:03,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-twwt.c.gcp-upstream.internal';"
2026-08-08 15:51:03,316 [I] CORE_DB_SCHEMA import status has been updated
2026-08-08 15:51:03,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-08 15:51:03,585 [I] DB_PATCHES have already been applied
I20260808 15:51:03.592314 27987 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-admin:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskzIk6P5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260808 15:51:03.592413 27987 mini_ranger.cc:193] Using host: 127.27.84.212
I20260808 15:51:03.615235 27987 mini_ranger.cc:227] Ranger admin URL: http://127.27.84.212:51659
Aug 08, 2026 3:51:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:51:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:51:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp
Aug 08, 2026 3:51:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 08, 2026 3:51:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 08, 2026 3:51:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-admin/ews/webapp].
Aug 08, 2026 3:51:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260808 15:51:05.036154 27987 mini_ranger.cc:160] Time spent starting Ranger: real 9.237s user 0.004s sys 0.068s
Aug 08, 2026 3:51:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 08, 2026 3:51:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 08, 2026 3:51:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 08, 2026 3:51:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 08, 2026 3:51:25 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260808 15:51:26.510360 27987 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.84.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.84.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/rangerkms_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.84.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/rangerkms_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.84.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.84.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.84.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/HTTP_127.27.84.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 08 15:51:26 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204286, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260808 15:51:26.740605 27987 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260808 15:51:26.769922 27987 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260808 15:51:27.717053 27987 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260808 15:51:27.717161 27987 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms
I20260808 15:51:27.717201 27987 mini_ranger_kms.cc:227] Using postgres at 127.27.84.212:43963
2026-08-08 15:51:28,098 [I] DB FLAVOR :POSTGRES
2026-08-08 15:51:28,099 [I] --------- Verifying Ranger DB connection ---------
2026-08-08 15:51:28,099 [I] Checking connection
2026-08-08 15:51:28,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-08 15:51:28,340 [I] connection success
2026-08-08 15:51:28,340 [I] --------- Verifying Ranger DB tables ---------
2026-08-08 15:51:28,340 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-08 15:51:28,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-08 15:51:28,613 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-08 15:51:28,613 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-08 15:51:28,613 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-08 15:51:28,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.84.212:43963/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-08 15:51:28,888 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260808 15:51:29.373301 27987 mini_ranger_kms.cc:310] Created kms service
I20260808 15:51:29.578193 27987 mini_ranger_kms.cc:326] Created kudu user
I20260808 15:51:29.623159 27987 mini_ranger_kms.cc:343] Created rangerkms user
Aug 08 15:51:29 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204286, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:30.049424 27987 mini_ranger_kms.cc:384] Added ranger policy
I20260808 15:51:30.049782 27987 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms:/tmp/dist-test-taskzIk6P5/build/asan/bin/postgresql.jar:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskzIk6P5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260808 15:51:30.049877 27987 mini_ranger_kms.cc:240] Using host: 127.27.84.212
I20260808 15:51:30.073073 27987 mini_ranger_kms.cc:276] Ranger KMS PID: 14816
I20260808 15:51:30.073266 27987 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.84.212:60313
Aug 08, 2026 3:51:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 08, 2026 3:51:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 08, 2026 3:51:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 08, 2026 3:51:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 08, 2026 3:51:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-kms/ews/webapp].
Aug 08, 2026 3:51:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 08 15:51:32 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.84.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:32.869351 27987 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.129s user 0.073s sys 0.346s
I20260808 15:51:32.869612 27987 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260808 15:51:32.869810 27987 mini_ranger_kms.cc:401] 127.27.84.212:60313/kms/v1/keys
Aug 08 15:51:32 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.84.212@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
Aug 08 15:51:33 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204292, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.84.212@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
Aug 08 15:51:35 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:51:35.703266 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34327
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:34327
--ranger_config_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:36.066350 14901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:36.066761 14901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:36.066942 14901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:36.074411 14901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:36.074554 14901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:36.074683 14901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:36.074785 14901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:36.074887 14901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:36.085117 14901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.84.212:60313/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:34327
--ranger_config_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:36.086477 14901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:36.087911 14901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:36.100546 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:36.100518 14920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:36.100521 14913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:36.100818 14901 server_base.cc:1034] running on GCE node
I20260808 15:51:36.101749 14901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:36.103271 14901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:36.104477 14901 hybrid_clock.cc:648] HybridClock initialized: now 1786204296104435 us; error 32 us; skew 500 ppm
I20260808 15:51:36.104846 14901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:36 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:36.108232 14901 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:51:36.109683 14901 webserver.cc:460] Webserver started at http://127.27.84.254:42609/ using document root <none> and password file <none>
I20260808 15:51:36.110306 14901 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:36.110405 14901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:36.110756 14901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:51:36 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:36.171303 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40"
format_stamp: "Formatted at 2026-08-08 15:51:36 on dist-test-slave-twwt"
server_key: "ea45a817f93c8326d77068a45e853605"
server_key_iv: "84edc78615aa3b78c872b0894896f9c7"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:36.172245 14901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40"
format_stamp: "Formatted at 2026-08-08 15:51:36 on dist-test-slave-twwt"
server_key: "ea45a817f93c8326d77068a45e853605"
server_key_iv: "84edc78615aa3b78c872b0894896f9c7"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:36.178265 14901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260808 15:51:36.200222 14932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:36.202114 14901 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:51:36.202333 14901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40"
format_stamp: "Formatted at 2026-08-08 15:51:36 on dist-test-slave-twwt"
server_key: "ea45a817f93c8326d77068a45e853605"
server_key_iv: "84edc78615aa3b78c872b0894896f9c7"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:36.202630 14901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:36.230217 14901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:36.234313 14901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:36.234594 14901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:36.254902 14901 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:34327
I20260808 15:51:36.254951 15013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:34327 every 8 connection(s)
I20260808 15:51:36.256356 14901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:51:36.259402 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 14901
I20260808 15:51:36.259624 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:51:36.260367 27987 external_mini_cluster.cc:1428] Setting key c06f823dd316a90cfd5a428e74af1c2f
I20260808 15:51:36.260403 15014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:36.275516 15014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Bootstrap starting.
Aug 08 15:51:36 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:36.279709 15014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:36.281111 15014 log.cc:824] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:36.284227 15014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: No bootstrap required, opened a new log
I20260808 15:51:36.285502 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:36.262498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.27.84.254:34327)
0808 15:51:36.263401 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:36.263417 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:36.263477 (+ 60us) server_negotiation.cc:407] Connection header received
0808 15:51:36.265493 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:36.265523 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:36.266054 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:36.266516 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:36.267898 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:36.269924 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:36.271352 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:36.272515 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:36.272728 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:36.277068 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:36.277080 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:36.277105 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:36.277177 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:36.281272 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:36.282043 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:36.282050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:36.282057 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:36.282250 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:36.282691 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:36.282695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:36.282698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:36.283214 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:36.283552 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:36.283998 (+ 446us) server_negotiation.cc:299] Negotiation successful
0808 15:51:36.284471 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":356,"threads_started":1}
I20260808 15:51:36.292078 15014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:36.292419 15014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:36.292511 15014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41cbfe5304d04ad9b7bae9a6fd7afd40, State: Initialized, Role: FOLLOWER
I20260808 15:51:36.293164 15014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:36.293483 15014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:51:36.293689 15014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:51:36.293962 15014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:36.296209 15014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:36.296705 15014 leader_election.cc:302] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41cbfe5304d04ad9b7bae9a6fd7afd40; no voters:
I20260808 15:51:36.297313 15014 leader_election.cc:288] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:51:36.297446 15022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:51:36.298183 15022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 LEADER]: Becoming Leader. State: Replica: 41cbfe5304d04ad9b7bae9a6fd7afd40, State: Running, Role: LEADER
I20260808 15:51:36.298849 15022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:36.299558 15014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:36.301867 15023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } } }
I20260808 15:51:36.301771 15025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41cbfe5304d04ad9b7bae9a6fd7afd40. Latest consensus state: current_term: 1 leader_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } } }
I20260808 15:51:36.302222 15023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:36.302295 15025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:36.303332 15033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:51:36.311779 15033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:51:36.323283 15014 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:51:36.326244 15033 catalog_manager.cc:1347] Generated new cluster ID: 4a3a002fe60b4335892f59324b59d554
I20260808 15:51:36.326371 15033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:51:36.380281 15033 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:51:36.382421 15033 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:51:36.392397 15033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Generated new TSK 0
I20260808 15:51:36.393559 15033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:51:37.138350 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34327
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:51:37 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260808 15:51:37.585104 15079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:37.585647 15079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:37.585855 15079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:37.597800 15079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:37.598050 15079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:37.598328 15079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:51:37.614478 15079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.84.212:60313/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:37.616667 15079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:37.618489 15079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:37.642592 15095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:37.644328 15096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:37.672643 15098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:37.673003 15079 server_base.cc:1034] running on GCE node
I20260808 15:51:37.673780 15079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:37.674921 15079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:37.676172 15079 hybrid_clock.cc:648] HybridClock initialized: now 1786204297676089 us; error 71 us; skew 500 ppm
I20260808 15:51:37.676523 15079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:37 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:37.679961 15079 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:51:37.681792 15079 webserver.cc:460] Webserver started at http://127.27.84.193:34831/ using document root <none> and password file <none>
I20260808 15:51:37.682564 15079 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:37.682758 15079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:37.683091 15079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:51:37 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:37.704103 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "b0d439ea757c48bf89bbd303dc6eaa0a"
format_stamp: "Formatted at 2026-08-08 15:51:37 on dist-test-slave-twwt"
server_key: "bbe7713b41d8b0a785a6d0828adaa495"
server_key_iv: "f795803e7c6179874411d769369c3e65"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:37.705476 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "b0d439ea757c48bf89bbd303dc6eaa0a"
format_stamp: "Formatted at 2026-08-08 15:51:37 on dist-test-slave-twwt"
server_key: "bbe7713b41d8b0a785a6d0828adaa495"
server_key_iv: "f795803e7c6179874411d769369c3e65"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:37.730382 15079 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.016s sys 0.006s
I20260808 15:51:37.761662 15108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:37.764200 15079 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.001s
I20260808 15:51:37.764441 15079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "b0d439ea757c48bf89bbd303dc6eaa0a"
format_stamp: "Formatted at 2026-08-08 15:51:37 on dist-test-slave-twwt"
server_key: "bbe7713b41d8b0a785a6d0828adaa495"
server_key_iv: "f795803e7c6179874411d769369c3e65"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:37.764782 15079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:37.792336 15079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:37.797953 15079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:37.798408 15079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:37.799772 15079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:37.803432 15079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:37.803558 15079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 15:51:37.803702 15079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:37.803781 15079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:37.878726 15079 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:37397
I20260808 15:51:37.878746 15284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:37397 every 8 connection(s)
I20260808 15:51:37.880493 15079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:51:37.885695 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 15079
I20260808 15:51:37.885941 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:51:37.886866 27987 external_mini_cluster.cc:1428] Setting key 91cd5b116bf29a8daf8cfaa8a0f08ebf
Aug 08 15:51:37 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
Aug 08 15:51:37 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:37.920219 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:37.885675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:46093 (local address 127.27.84.254:34327)
0808 15:51:37.895118 (+ 9443us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:37.895125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:37.898664 (+ 3539us) server_negotiation.cc:407] Connection header received
0808 15:51:37.899050 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:37.899059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:37.899166 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:37.899253 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:37.901752 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:37.903017 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:37.905770 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:37.906539 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:37.906693 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:37.912286 (+ 5593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:37.912294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:37.912299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:37.912345 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:37.916106 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:37.917150 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:37.917157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:37.917161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:37.917354 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:37.917931 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:37.917937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:37.917940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:37.918307 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:37.918431 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:37.919553 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0808 15:51:37.919802 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9303,"thread_start_us":9186,"threads_started":1}
I20260808 15:51:37.922698 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:37.884858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34327 (local address 127.27.84.193:46093)
0808 15:51:37.895714 (+ 10856us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:37.895781 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:37.898516 (+ 2735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:37.899889 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:37.899905 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:37.900602 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:37.901524 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:37.901557 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:37.903250 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:37.903258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:37.905587 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:37.905606 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:37.907086 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:37.907096 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:37.907407 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:37.908851 (+ 1444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:37.908887 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:37.911981 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:37.916309 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:37.916320 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:37.916340 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:37.916996 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:37.917515 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:37.917520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:37.917523 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:37.917794 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:37.918551 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:37.918560 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:37.919066 (+ 506us) client_negotiation.cc:769] Sending connection context
0808 15:51:37.920582 (+ 1516us) client_negotiation.cc:240] Negotiation successful
0808 15:51:37.921189 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10507,"thread_start_us":22411,"threads_started":1}
I20260808 15:51:37.927830 15286 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34327
I20260808 15:51:37.928234 15286 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:37.929168 15286 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:37.931949 14959 ts_manager.cc:194] Registered new tserver with Master: b0d439ea757c48bf89bbd303dc6eaa0a (127.27.84.193:37397)
I20260808 15:51:37.934239 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:46093
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
W20260808 15:51:38.007820 27987 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.84.194: real 0.111s user 0.000s sys 0.027s
I20260808 15:51:38.008946 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34327
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:51:38 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
W20260808 15:51:38.527354 15296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:38.527906 15296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:38.528111 15296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:38.539849 15296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:38.540112 15296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:38.540361 15296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:51:38.556358 15296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.84.212:60313/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:38.558494 15296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:38.560400 15296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:38.603497 15310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:38.607182 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:38.614442 15296 server_base.cc:1034] running on GCE node
W20260808 15:51:38.614959 15312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:38.615917 15296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:38.629407 15296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:38.630771 15296 hybrid_clock.cc:648] HybridClock initialized: now 1786204298630684 us; error 89 us; skew 500 ppm
I20260808 15:51:38.631341 15296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:38 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:38.636080 15296 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:51:38.639334 15296 webserver.cc:460] Webserver started at http://127.27.84.194:37235/ using document root <none> and password file <none>
I20260808 15:51:38.640147 15296 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:38.640307 15296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:38.640753 15296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:51:38 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:38.662925 15296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "e5111269974c411d9baaf2a72ca6d3b4"
format_stamp: "Formatted at 2026-08-08 15:51:38 on dist-test-slave-twwt"
server_key: "bc3152afb05852415d540a0817b78439"
server_key_iv: "ce130785aed4d91386ffec02dfa52454"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:38.664017 15296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "e5111269974c411d9baaf2a72ca6d3b4"
format_stamp: "Formatted at 2026-08-08 15:51:38 on dist-test-slave-twwt"
server_key: "bc3152afb05852415d540a0817b78439"
server_key_iv: "ce130785aed4d91386ffec02dfa52454"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:38.671521 15296 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260808 15:51:38.691241 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:38.692938 15296 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 15:51:38.693137 15296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "e5111269974c411d9baaf2a72ca6d3b4"
format_stamp: "Formatted at 2026-08-08 15:51:38 on dist-test-slave-twwt"
server_key: "bc3152afb05852415d540a0817b78439"
server_key_iv: "ce130785aed4d91386ffec02dfa52454"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:38.693399 15296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:38.719048 15296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:38.723436 15296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:38.723778 15296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:38.724841 15296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:38.726411 15296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:38.726516 15296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:38.726687 15296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:38.726773 15296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:38.762212 15296 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:38157
I20260808 15:51:38.762259 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:38157 every 8 connection(s)
I20260808 15:51:38.763684 15296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:51:38.770144 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 15296
I20260808 15:51:38.770370 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:51:38.771008 27987 external_mini_cluster.cc:1428] Setting key 961b78859a72786b777e20223d9dae13
Aug 08 15:51:38 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:38.786375 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:38.766713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:50511 (local address 127.27.84.254:34327)
0808 15:51:38.767275 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:38.767280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:38.768177 (+ 897us) server_negotiation.cc:407] Connection header received
0808 15:51:38.770246 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:38.770254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:38.770388 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:38.770528 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:38.772528 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:38.773496 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:38.775403 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:38.775892 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:38.776059 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:38.780147 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:38.780152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:38.780156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:38.780191 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:38.783094 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:38.783903 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:38.783908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:38.783910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:38.784039 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:38.784516 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:38.784520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:38.784522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:38.784818 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:38.784984 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:38.785735 (+ 751us) server_negotiation.cc:299] Negotiation successful
0808 15:51:38.785984 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":269,"threads_started":1}
I20260808 15:51:38.787423 15522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:38.767179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34327 (local address 127.27.84.194:50511)
0808 15:51:38.767995 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:38.768040 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:38.769943 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:38.770841 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:38.770851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:38.771455 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:38.772337 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:38.772360 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:38.773678 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:38.773689 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:38.775138 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:38.775152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:38.776168 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:38.776177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:38.776563 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:38.777348 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:38.777373 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:38.779882 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:38.783262 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:38.783272 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:38.783291 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:38.783754 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:38.784159 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:38.784164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:38.784168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:38.784378 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:38.784959 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:38.784967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:38.785335 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 15:51:38.785609 (+ 274us) client_negotiation.cc:240] Negotiation successful
0808 15:51:38.786172 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":322,"threads_started":1}
I20260808 15:51:38.789032 15519 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34327
I20260808 15:51:38.789366 15519 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:38.790088 15519 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:38.791751 14959 ts_manager.cc:194] Registered new tserver with Master: e5111269974c411d9baaf2a72ca6d3b4 (127.27.84.194:38157)
I20260808 15:51:38.793071 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:50511
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:51:38.837865 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34327
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:51:38.938143 15286 heartbeater.cc:503] Master 127.27.84.254:34327 was elected leader, sending a full tablet report...
W20260808 15:51:39.187669 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:39.188072 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:39.188256 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:39.195768 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:39.195946 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:39.196125 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:51:39.206775 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.84.212:60313/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:39.208271 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:39.209681 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:39.224689 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:39.224826 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:39.224864 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:39.226523 15528 server_base.cc:1034] running on GCE node
I20260808 15:51:39.227332 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:39.228425 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:39.229689 15528 hybrid_clock.cc:648] HybridClock initialized: now 1786204299229631 us; error 47 us; skew 500 ppm
I20260808 15:51:39.230144 15528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:39 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:39.238999 15528 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:51:39.240916 15528 webserver.cc:460] Webserver started at http://127.27.84.195:37559/ using document root <none> and password file <none>
I20260808 15:51:39.241667 15528 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:39.241819 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:39.242224 15528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 08 15:51:39 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:39.260950 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "29ebf1ee19c245c582929a611fa5e311"
format_stamp: "Formatted at 2026-08-08 15:51:39 on dist-test-slave-twwt"
server_key: "a1f6772cc9f6d0bd9e65b094538b88fc"
server_key_iv: "18f968ab6988a6a8a00e4af9ab550fbd"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:39.262041 15528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "29ebf1ee19c245c582929a611fa5e311"
format_stamp: "Formatted at 2026-08-08 15:51:39 on dist-test-slave-twwt"
server_key: "a1f6772cc9f6d0bd9e65b094538b88fc"
server_key_iv: "18f968ab6988a6a8a00e4af9ab550fbd"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:39.268570 15528 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260808 15:51:39.285152 15548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:39.286863 15528 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:51:39.287109 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "29ebf1ee19c245c582929a611fa5e311"
format_stamp: "Formatted at 2026-08-08 15:51:39 on dist-test-slave-twwt"
server_key: "a1f6772cc9f6d0bd9e65b094538b88fc"
server_key_iv: "18f968ab6988a6a8a00e4af9ab550fbd"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:39.287348 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:39.311538 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:39.316037 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:39.316507 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:39.317669 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:39.319439 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:39.319545 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:39.319720 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:39.319861 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:39.375519 15528 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:35907
I20260808 15:51:39.375566 15721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:35907 every 8 connection(s)
I20260808 15:51:39.377115 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:51:39.379469 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 15528
I20260808 15:51:39.379679 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:51:39.380414 27987 external_mini_cluster.cc:1428] Setting key 8bdc5d06e3dcfa97b44f9abe79a1a2d6
Aug 08 15:51:39 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:39.402915 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:39.381821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:33535 (local address 127.27.84.254:34327)
0808 15:51:39.382547 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:39.382555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:39.383838 (+ 1283us) server_negotiation.cc:407] Connection header received
0808 15:51:39.386505 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:39.386511 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:39.386627 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:39.386770 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:39.388419 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.389165 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.391070 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.391599 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.391741 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:39.395640 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:39.395645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:39.395648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:39.395680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:39.399611 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.400482 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.400486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.400489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.400670 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.401204 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.401208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.401210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.401487 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:39.401611 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:39.402299 (+ 688us) server_negotiation.cc:299] Negotiation successful
0808 15:51:39.402475 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":384,"threads_started":1}
I20260808 15:51:39.403976 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:39.382518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34327 (local address 127.27.84.195:33535)
0808 15:51:39.383675 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:39.383714 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:39.386223 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:39.387017 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:39.387025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:39.387571 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:39.388233 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:39.388262 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.389346 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.389350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:39.390924 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:39.390937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.391885 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.391895 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:39.392117 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:39.393010 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:39.393033 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:39.395376 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:39.399735 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:39.399742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:39.399755 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:39.400356 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:39.400802 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:39.400806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:39.400809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:39.401082 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:39.401634 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:39.401639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:39.401955 (+ 316us) client_negotiation.cc:769] Sending connection context
0808 15:51:39.402185 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 15:51:39.402553 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":630,"threads_started":1}
I20260808 15:51:39.405470 15722 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34327
I20260808 15:51:39.405819 15722 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:39.406661 15722 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:39.408394 14959 ts_manager.cc:194] Registered new tserver with Master: 29ebf1ee19c245c582929a611fa5e311 (127.27.84.195:35907)
I20260808 15:51:39.409749 14959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:33535
I20260808 15:51:39.411790 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:51:39.413518 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 15079
Aug 08 15:51:39 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:51:39.434895 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:39.414761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.27.84.193:37397)
0808 15:51:39.415446 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:39.415453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:39.415477 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:51:39.415535 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:39.415540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:39.415715 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:39.415888 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:39.418158 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.419728 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.421540 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.422250 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.422364 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:39.428226 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:39.428239 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:39.428253 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:39.428342 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:39.431712 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.432465 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.432473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.432475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.432632 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.433096 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.433101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.433107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.433465 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:39.433877 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:39.434170 (+ 293us) server_negotiation.cc:299] Negotiation successful
0808 15:51:39.434486 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":373,"threads_started":1}
I20260808 15:51:39.436184 15273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:39.525625 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 15079
I20260808 15:51:39.550495 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 15296
Aug 08 15:51:39 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:51:39.572847 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:39.551479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.27.84.194:38157)
0808 15:51:39.552232 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:39.552239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:39.552291 (+ 52us) server_negotiation.cc:407] Connection header received
0808 15:51:39.552356 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:39.552362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:39.552595 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:39.552745 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:39.554031 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.555255 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.556679 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.557250 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.557433 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:39.566741 (+ 9308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:39.566758 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:39.566776 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:39.566817 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:39.569941 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.570684 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.570693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.570695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.570845 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.571315 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.571318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.571321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.571679 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:39.571938 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:39.572294 (+ 356us) server_negotiation.cc:299] Negotiation successful
0808 15:51:39.572517 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":482,"threads_started":1}
I20260808 15:51:39.574268 15510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:39.657666 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 15296
I20260808 15:51:39.680632 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 15528
Aug 08 15:51:39 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:51:39.697552 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:39.681614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.27.84.195:35907)
0808 15:51:39.682311 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:39.682318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:39.682346 (+ 28us) server_negotiation.cc:407] Connection header received
0808 15:51:39.682410 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:39.682417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:39.682700 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:39.682830 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:39.684140 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.685146 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.686372 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:39.687105 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:39.687203 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:39.691676 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:39.691693 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:39.691716 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:39.691752 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:39.694903 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.695552 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.695559 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.695561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.695687 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:39.696089 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:39.696092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:39.696094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:39.696424 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:39.696655 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:39.696936 (+ 281us) server_negotiation.cc:299] Negotiation successful
0808 15:51:39.697131 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":370,"threads_started":1}
I20260808 15:51:39.699013 15712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:39.784646 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 15528
I20260808 15:51:39.808796 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 14901
I20260808 15:51:39.810211 15005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:39.921739 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 14901
I20260808 15:51:39.955125 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34327
--webserver_interface=127.27.84.254
--webserver_port=42609
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:34327
--ranger_config_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:40.303917 15741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:40.304311 15741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:40.304520 15741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:40.311849 15741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:40.311960 15741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:40.312071 15741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:40.312150 15741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:40.312239 15741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:40.322698 15741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:34327
--ranger_config_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=42609
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:40.324019 15741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:40.325461 15741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:40.337846 15753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:40.337913 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:40.337931 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:40.338997 15741 server_base.cc:1034] running on GCE node
I20260808 15:51:40.339769 15741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:40.341149 15741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:40.342392 15741 hybrid_clock.cc:648] HybridClock initialized: now 1786204300342344 us; error 35 us; skew 500 ppm
I20260808 15:51:40.342869 15741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:40 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:40.346053 15741 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:51:40.347563 15741 webserver.cc:460] Webserver started at http://127.27.84.254:42609/ using document root <none> and password file <none>
I20260808 15:51:40.348176 15741 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:40.348289 15741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:51:40 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204300, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:40.435187 15741 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260808 15:51:40.438342 15775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:40.439780 15741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:51:40.439983 15741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40"
format_stamp: "Formatted at 2026-08-08 15:51:36 on dist-test-slave-twwt"
server_key: "ea45a817f93c8326d77068a45e853605"
server_key_iv: "84edc78615aa3b78c872b0894896f9c7"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:40.440989 15741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:40.457557 15741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:40.462031 15741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:40.462342 15741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:40.482016 15741 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:34327
I20260808 15:51:40.481997 15869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:34327 every 8 connection(s)
I20260808 15:51:40.483583 15741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:51:40.487547 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 15741
I20260808 15:51:40.488497 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:37397
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:34327
--webserver_port=34831
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:51:40.491055 15870 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:51:40.494391 15870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Bootstrap starting.
I20260808 15:51:40.517540 15870 log.cc:824] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:40.526536 15870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:51:40.527163 15870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Bootstrap complete.
I20260808 15:51:40.536739 15870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:40.537643 15870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41cbfe5304d04ad9b7bae9a6fd7afd40, State: Initialized, Role: FOLLOWER
I20260808 15:51:40.538350 15870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:40.538589 15870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:51:40.538805 15870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:51:40.539007 15870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:51:40.542934 15870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:40.543450 15870 leader_election.cc:302] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41cbfe5304d04ad9b7bae9a6fd7afd40; no voters:
I20260808 15:51:40.544108 15870 leader_election.cc:288] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:51:40.544292 15877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:51:40.546747 15870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:40.550916 15877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [term 2 LEADER]: Becoming Leader. State: Replica: 41cbfe5304d04ad9b7bae9a6fd7afd40, State: Running, Role: LEADER
I20260808 15:51:40.551642 15877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } }
I20260808 15:51:40.554916 15880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41cbfe5304d04ad9b7bae9a6fd7afd40. Latest consensus state: current_term: 2 leader_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } } }
I20260808 15:51:40.555291 15880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:40.554384 15878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41cbfe5304d04ad9b7bae9a6fd7afd40" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34327 } } }
I20260808 15:51:40.555797 15878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:40.559508 15900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:51:40.565219 15900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:51:40.566277 15870 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260808 15:51:40.566654 15900 catalog_manager.cc:1259] Loaded cluster ID: 4a3a002fe60b4335892f59324b59d554
I20260808 15:51:40.566749 15900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:51:40.570820 15900 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:51:40.572229 15900 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Loaded TSK: 0
I20260808 15:51:40.573765 15900 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:51:40.977903 15874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:40.978343 15874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:40.978523 15874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:40.990247 15874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:40.990420 15874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:40.990666 15874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:51:41.007289 15874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:37397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=34831
--tserver_master_addrs=127.27.84.254:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:41.009301 15874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:41.011379 15874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:41.032859 15936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:41.033582 15937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:41.034992 15941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:41.035465 15874 server_base.cc:1034] running on GCE node
I20260808 15:51:41.036134 15874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:41.037256 15874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:41.038461 15874 hybrid_clock.cc:648] HybridClock initialized: now 1786204301038406 us; error 49 us; skew 500 ppm
I20260808 15:51:41.038894 15874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:41.042582 15874 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:51:41.051934 15874 webserver.cc:460] Webserver started at http://127.27.84.193:34831/ using document root <none> and password file <none>
I20260808 15:51:41.052798 15874 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:41.052953 15874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:41.079329 15874 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260808 15:51:41.086125 15958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:41.088228 15874 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260808 15:51:41.088450 15874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "b0d439ea757c48bf89bbd303dc6eaa0a"
format_stamp: "Formatted at 2026-08-08 15:51:37 on dist-test-slave-twwt"
server_key: "bbe7713b41d8b0a785a6d0828adaa495"
server_key_iv: "f795803e7c6179874411d769369c3e65"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:41.089969 15874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:41.126681 15874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:41.132404 15874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:41.132763 15874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:41.134326 15874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:41.136500 15874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:41.136615 15874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260808 15:51:41.136723 15874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:41.136806 15874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:41.222642 15874 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:37397
I20260808 15:51:41.222711 16131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:37397 every 8 connection(s)
I20260808 15:51:41.224462 15874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:51:41.228761 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 15874
I20260808 15:51:41.229853 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:38157
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:34327
--webserver_port=37235
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:41.265456 16134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:41.228172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34327 (local address 127.27.84.193:39391)
0808 15:51:41.236131 (+ 7959us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:41.236205 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:41.239020 (+ 2815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:41.241604 (+ 2584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:41.241618 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:41.242391 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:41.243295 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:41.243325 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.246115 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.246123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:41.248437 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:41.248457 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.249769 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.249777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:41.250079 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:41.253543 (+ 3464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:41.253594 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:41.255888 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:41.260113 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:41.260124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:41.260142 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:41.260628 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:41.261147 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:41.261152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:41.261156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:41.261396 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:41.262601 (+ 1205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:41.262610 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:41.263035 (+ 425us) client_negotiation.cc:769] Sending connection context
0808 15:51:41.263336 (+ 301us) client_negotiation.cc:240] Negotiation successful
0808 15:51:41.263962 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7606,"thread_start_us":7333,"threads_started":1}
I20260808 15:51:41.265443 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:41.229365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:39391 (local address 127.27.84.254:34327)
0808 15:51:41.230759 (+ 1394us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:41.230782 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:41.236699 (+ 5917us) server_negotiation.cc:407] Connection header received
0808 15:51:41.239911 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:41.239933 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:41.240458 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:41.240869 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:41.243576 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.245870 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.248633 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.249478 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.249722 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:41.256224 (+ 6502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:41.256233 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:41.256250 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:41.256302 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:41.259940 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:41.260813 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:41.260828 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:41.260834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:41.261015 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:41.261540 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:41.261544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:41.261547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:41.262305 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:41.262518 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:41.263599 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0808 15:51:41.263990 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1062,"thread_start_us":862,"threads_started":1}
I20260808 15:51:41.269016 16132 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34327
I20260808 15:51:41.269346 16132 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:41.270138 16132 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
W20260808 15:51:41.273404 16132 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:34327 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:51:41.273653 16132 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:41.273938 16132 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:41.275267 16132 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:41.275537 16132 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
W20260808 15:51:41.276602 16132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:51:41.493920 16147 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260808 15:51:41.677834 16138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:41.678161 16138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:41.678323 16138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:41.685849 16138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:41.685985 16138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:41.686165 16138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:51:41.701603 16138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:38157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=37235
--tserver_master_addrs=127.27.84.254:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:41.703486 16138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:41.705243 16138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:41.727082 16166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:41.727957 16163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:41.729548 16168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:41.730719 16138 server_base.cc:1034] running on GCE node
I20260808 15:51:41.731650 16138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:41.733106 16138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:41.734390 16138 hybrid_clock.cc:648] HybridClock initialized: now 1786204301734309 us; error 60 us; skew 500 ppm
I20260808 15:51:41.734869 16138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:41.739511 16138 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:51:41.741385 16138 webserver.cc:460] Webserver started at http://127.27.84.194:37235/ using document root <none> and password file <none>
I20260808 15:51:41.742183 16138 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:41.742364 16138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:41.777841 16138 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.004s
I20260808 15:51:41.791051 16184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:41.793339 16138 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.009s
I20260808 15:51:41.793562 16138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "e5111269974c411d9baaf2a72ca6d3b4"
format_stamp: "Formatted at 2026-08-08 15:51:38 on dist-test-slave-twwt"
server_key: "bc3152afb05852415d540a0817b78439"
server_key_iv: "ce130785aed4d91386ffec02dfa52454"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:41.795145 16138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:41.826850 16138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:41.832453 16138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:41.832803 16138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:41.834142 16138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:41.836031 16138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:41.836145 16138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:41.836261 16138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:41.836342 16138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:41.895478 16138 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:38157
I20260808 15:51:41.895535 16357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:38157 every 8 connection(s)
I20260808 15:51:41.897297 16138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:51:41.907240 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 16138
I20260808 15:51:41.908294 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.84.212:60313/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:35907
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:34327
--webserver_port=37559
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:40435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:51:41 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:41.939888 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:41.902680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34327 (local address 127.27.84.194:50701)
0808 15:51:41.914024 (+ 11344us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:41.914092 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:41.918669 (+ 4577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:41.919964 (+ 1295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:41.919980 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:41.920725 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:41.921632 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:41.921663 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.923383 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.923391 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:41.924715 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:41.924728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.926191 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.926202 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:41.926545 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:41.927429 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:41.927465 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:41.930480 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:41.934692 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:41.934702 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:41.934722 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:41.935384 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:41.935891 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:41.935897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:41.935901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:41.936178 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:41.936944 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:41.936953 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:41.937429 (+ 476us) client_negotiation.cc:769] Sending connection context
0808 15:51:41.937743 (+ 314us) client_negotiation.cc:240] Negotiation successful
0808 15:51:41.938354 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10924,"thread_start_us":11830,"threads_started":1}
I20260808 15:51:41.941056 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:41.902006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:50701 (local address 127.27.84.254:34327)
0808 15:51:41.917235 (+ 15229us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:41.917243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:41.917273 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:51:41.919195 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:41.919204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:41.919313 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:41.919427 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:41.921913 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.923155 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.924881 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:41.925672 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:41.925816 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:41.930802 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:41.930810 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:41.930815 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:41.930860 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:41.934462 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:41.935565 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:41.935571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:41.935575 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:41.935748 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:41.936333 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:41.936338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:41.936342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:41.936710 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:41.936866 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:41.940456 (+ 3590us) server_negotiation.cc:299] Negotiation successful
0808 15:51:41.940695 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15064,"thread_start_us":15270,"threads_started":1}
I20260808 15:51:41.942485 16358 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34327
I20260808 15:51:41.942883 16358 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:41.943812 16358 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:41.946067 15813 ts_manager.cc:194] Registered new tserver with Master: e5111269974c411d9baaf2a72ca6d3b4 (127.27.84.194:38157)
I20260808 15:51:41.948451 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:50701
I20260808 15:51:42.276786 16132 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:42.277294 16132 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:42.278697 15813 ts_manager.cc:194] Registered new tserver with Master: b0d439ea757c48bf89bbd303dc6eaa0a (127.27.84.193:37397)
I20260808 15:51:42.279906 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:39391
Aug 08 15:51:42 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
W20260808 15:51:42.370707 16362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:42.371148 16362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:42.371312 16362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:42.382840 16362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:42.383040 16362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:42.383247 16362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:51:42.400126 16362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:40435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.84.212:60313/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:35907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=37559
--tserver_master_addrs=127.27.84.254:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:42.401947 16362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:42.403838 16362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:42.464453 16374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:42.464547 16376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:42.472164 16380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:42.472759 16362 server_base.cc:1034] running on GCE node
I20260808 15:51:42.473618 16362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:42.474947 16362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:42.476200 16362 hybrid_clock.cc:648] HybridClock initialized: now 1786204302476121 us; error 63 us; skew 500 ppm
I20260808 15:51:42.476599 16362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:42 dist-test-slave-twwt krb5kdc[14198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204302, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:42.481104 16362 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:51:42.492722 16362 webserver.cc:460] Webserver started at http://127.27.84.195:37559/ using document root <none> and password file <none>
I20260808 15:51:42.493769 16362 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:42.493947 16362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 08 15:51:42 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204302, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:42.516929 16362 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:51:42.520995 16388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:42.523025 16362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:51:42.523234 16362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "29ebf1ee19c245c582929a611fa5e311"
format_stamp: "Formatted at 2026-08-08 15:51:39 on dist-test-slave-twwt"
server_key: "a1f6772cc9f6d0bd9e65b094538b88fc"
server_key_iv: "18f968ab6988a6a8a00e4af9ab550fbd"
server_key_version: "kuduclusterkey@0"
I20260808 15:51:42.524700 16362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:42.543633 16362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:42.548552 16362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:42.548849 16362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:42.550048 16362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:42.551734 16362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:42.551858 16362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:42.552006 16362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:42.552098 16362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:42.809762 16362 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:35907
I20260808 15:51:42.809809 16569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:35907 every 8 connection(s)
I20260808 15:51:42.811813 16362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:51:42.821411 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 16362
Aug 08 15:51:42 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204301, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for HTTP/127.27.84.212@KRBTEST.COM
I20260808 15:51:42.874862 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:42.822532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.27.84.254:34327)
0808 15:51:42.835500 (+ 12968us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:42.835508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:42.835532 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:51:42.835605 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:42.835613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:42.835750 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:42.835885 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:42.838190 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.839414 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.844301 (+ 4887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.845257 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.845396 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:42.864055 (+ 18659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:42.864063 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:42.864068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:42.864129 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:42.868044 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:42.869430 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:42.869439 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:42.869444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:42.869613 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:42.872936 (+ 3323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:42.872942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:42.872946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:42.873360 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:42.873676 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:42.874170 (+ 494us) server_negotiation.cc:299] Negotiation successful
0808 15:51:42.874437 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12836,"thread_start_us":12723,"threads_started":1}
Aug 08 15:51:42 dist-test-slave-twwt krb5kdc[14198](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204302, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:42.895081 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:42.835475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:50813 (local address 127.27.84.254:34327)
0808 15:51:42.836426 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:42.836432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:42.860562 (+ 24130us) server_negotiation.cc:407] Connection header received
0808 15:51:42.867791 (+ 7229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:42.867800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:42.867969 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:42.868075 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:42.873264 (+ 5189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.874523 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.878569 (+ 4046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.879232 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.879335 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:42.884020 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:42.884028 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:42.884033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:42.884080 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:42.887891 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:42.891771 (+ 3880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:42.891777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:42.891781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:42.891945 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:42.892611 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:42.892617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:42.892620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:42.893027 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:42.893169 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:42.894346 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0808 15:51:42.894686 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":667,"threads_started":1}
I20260808 15:51:42.897128 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:42.828302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34327 (local address 127.27.84.195:50813)
0808 15:51:42.860306 (+ 32004us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:42.860395 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:42.867300 (+ 6905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:42.870039 (+ 2739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:42.870058 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:42.870988 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:42.872143 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:42.872178 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.874949 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.874953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:42.878382 (+ 3429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:42.878401 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.879739 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.879745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:42.880007 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:42.880781 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:42.880806 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:42.883673 (+ 2867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:42.888098 (+ 4425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:42.888108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:42.888129 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:42.891617 (+ 3488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:42.892117 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:42.892122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:42.892126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:42.892462 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:42.893294 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:42.893304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:42.893829 (+ 525us) client_negotiation.cc:769] Sending connection context
0808 15:51:42.895392 (+ 1563us) client_negotiation.cc:240] Negotiation successful
0808 15:51:42.896043 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31504,"thread_start_us":32137,"threads_started":1}
I20260808 15:51:42.898700 16570 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34327
I20260808 15:51:42.899082 16570 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:42.899925 16570 heartbeater.cc:511] Master 127.27.84.254:34327 requested a full tablet report, sending...
I20260808 15:51:42.901630 15813 ts_manager.cc:194] Registered new tserver with Master: 29ebf1ee19c245c582929a611fa5e311 (127.27.84.195:35907)
I20260808 15:51:42.902758 15813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:50813
I20260808 15:51:42.905625 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:51:42.953713 16358 heartbeater.cc:503] Master 127.27.84.254:34327 was elected leader, sending a full tablet report...
I20260808 15:51:42.984887 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:42.947363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.27.84.254:34327)
0808 15:51:42.947569 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:42.947576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:42.962316 (+ 14740us) server_negotiation.cc:407] Connection header received
0808 15:51:42.963000 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:42.963007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:42.963109 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:42.963183 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:42.965790 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.966957 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.969091 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:42.969809 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:42.969937 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:42.977440 (+ 7503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:42.977448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:42.977453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:42.977501 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:42.981264 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:42.982220 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:42.982226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:42.982230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:42.982405 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:42.982987 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:42.982993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:42.982996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:42.983381 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:42.983606 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:42.984248 (+ 642us) server_negotiation.cc:299] Negotiation successful
0808 15:51:42.984510 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260808 15:51:42.992470 15813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:51:43.004974 15813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:51:43.092932 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.051956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.27.84.193:37397)
0808 15:51:43.077003 (+ 25047us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.077014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.078319 (+ 1305us) server_negotiation.cc:407] Connection header received
0808 15:51:43.078508 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.078517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.078854 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.079077 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:43.081606 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.083354 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.087699 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.089291 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.089412 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.089660 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.092395 (+ 2735us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.092578 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24813,"thread_start_us":25038,"threads_started":1}
I20260808 15:51:43.090893 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.051740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60230)
0808 15:51:43.078136 (+ 26396us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.078157 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:43.078344 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:43.079398 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:43.079406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:43.079458 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:43.081370 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.081384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.083574 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.083580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.087512 (+ 3932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.087528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.089870 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.089876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.090090 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.090142 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 15:51:43.090292 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 15:51:43.090505 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26268,"thread_start_us":26118,"threads_started":1}
I20260808 15:51:43.096182 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.050499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:35907 (local address 127.0.0.1:38954)
0808 15:51:43.075930 (+ 25431us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.076012 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:43.076302 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:43.079879 (+ 3577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:43.079884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:43.079912 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:43.080346 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.080361 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.085947 (+ 5586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.085953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.088129 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.088145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.095229 (+ 7084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.095236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.095519 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.095554 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:51:43.095633 (+ 79us) client_negotiation.cc:240] Negotiation successful
0808 15:51:43.095772 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24997,"thread_start_us":35126,"threads_started":1}
I20260808 15:51:43.097733 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.050873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38954 (local address 127.27.84.195:35907)
0808 15:51:43.078936 (+ 28063us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.078943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.078973 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:51:43.079072 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.079080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.079335 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.079536 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:43.081495 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.083236 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.088362 (+ 5126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.089902 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.096840 (+ 6938us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.097137 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.097214 (+ 77us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.097400 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27889,"thread_start_us":27752,"threads_started":1}
I20260808 15:51:43.097784 16035 tablet_service.cc:1467] Processing CreateTablet for tablet 60826a8ab3ac4c409c4aa8b98372349e (DEFAULT_TABLE table=test-table [id=963e895f9fe94c5cb6bf84aef610f4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:43.100883 16035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60826a8ab3ac4c409c4aa8b98372349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:43.103452 16465 tablet_service.cc:1467] Processing CreateTablet for tablet 60826a8ab3ac4c409c4aa8b98372349e (DEFAULT_TABLE table=test-table [id=963e895f9fe94c5cb6bf84aef610f4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:43.105417 16465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60826a8ab3ac4c409c4aa8b98372349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:43.121565 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.056525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.27.84.194:38157)
0808 15:51:43.077697 (+ 21172us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.077706 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.085606 (+ 7900us) server_negotiation.cc:407] Connection header received
0808 15:51:43.090362 (+ 4756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.090373 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.090722 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.090946 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:43.091935 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.094067 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.118248 (+ 24181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.119886 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.120014 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.120265 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.121001 (+ 736us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.121195 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20990,"thread_start_us":20814,"threads_started":1}
I20260808 15:51:43.122184 16605 tablet_bootstrap.cc:492] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Bootstrap starting.
I20260808 15:51:43.122802 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.055796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:38157 (local address 127.0.0.1:42554)
0808 15:51:43.085457 (+ 29661us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.085483 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:43.090111 (+ 4628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:43.091279 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:43.091284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:43.091313 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:43.091697 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.091711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.094255 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.094260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.111167 (+ 16907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.111191 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.120512 (+ 9321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.120521 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.120727 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.120764 (+ 37us) client_negotiation.cc:769] Sending connection context
0808 15:51:43.122150 (+ 1386us) client_negotiation.cc:240] Negotiation successful
0808 15:51:43.122405 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29503,"thread_start_us":19484,"threads_started":1}
I20260808 15:51:43.126176 16605 tablet_bootstrap.cc:654] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:43.127581 16605 log.cc:824] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:43.127345 16260 tablet_service.cc:1467] Processing CreateTablet for tablet 60826a8ab3ac4c409c4aa8b98372349e (DEFAULT_TABLE table=test-table [id=963e895f9fe94c5cb6bf84aef610f4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:43.129102 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60826a8ab3ac4c409c4aa8b98372349e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:43.136036 16607 tablet_bootstrap.cc:492] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Bootstrap starting.
I20260808 15:51:43.141819 16607 tablet_bootstrap.cc:654] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:43.143316 16607 log.cc:824] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:43.147886 16605 tablet_bootstrap.cc:492] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: No bootstrap required, opened a new log
I20260808 15:51:43.148218 16605 ts_tablet_manager.cc:1397] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.017s
I20260808 15:51:43.151280 16607 tablet_bootstrap.cc:492] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: No bootstrap required, opened a new log
I20260808 15:51:43.151578 16607 ts_tablet_manager.cc:1397] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260808 15:51:43.160697 16605 raft_consensus.cc:348] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.161258 16605 raft_consensus.cc:374] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:43.161392 16605 raft_consensus.cc:729] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d439ea757c48bf89bbd303dc6eaa0a, State: Initialized, Role: FOLLOWER
I20260808 15:51:43.162214 16605 consensus_queue.cc:260] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.163518 16610 tablet_bootstrap.cc:492] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Bootstrap starting.
I20260808 15:51:43.169106 16610 tablet_bootstrap.cc:654] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:43.170511 16610 log.cc:824] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:43.172399 16132 heartbeater.cc:503] Master 127.27.84.254:34327 was elected leader, sending a full tablet report...
I20260808 15:51:43.172421 16607 raft_consensus.cc:348] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.172945 16607 raft_consensus.cc:374] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:43.173046 16607 raft_consensus.cc:729] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ebf1ee19c245c582929a611fa5e311, State: Initialized, Role: FOLLOWER
I20260808 15:51:43.173815 16607 consensus_queue.cc:260] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.181128 16605 ts_tablet_manager.cc:1428] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Time spent starting tablet: real 0.033s user 0.007s sys 0.009s
I20260808 15:51:43.181110 16570 heartbeater.cc:503] Master 127.27.84.254:34327 was elected leader, sending a full tablet report...
I20260808 15:51:43.182176 16607 ts_tablet_manager.cc:1428] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Time spent starting tablet: real 0.030s user 0.013s sys 0.007s
I20260808 15:51:43.190260 16610 tablet_bootstrap.cc:492] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: No bootstrap required, opened a new log
I20260808 15:51:43.190634 16610 ts_tablet_manager.cc:1397] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.007s
I20260808 15:51:43.203540 16610 raft_consensus.cc:348] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.204182 16610 raft_consensus.cc:374] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:43.204392 16610 raft_consensus.cc:729] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5111269974c411d9baaf2a72ca6d3b4, State: Initialized, Role: FOLLOWER
I20260808 15:51:43.205256 16610 consensus_queue.cc:260] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.207628 16610 ts_tablet_manager.cc:1428] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
W20260808 15:51:43.234113 16133 tablet.cc:2369] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:51:43.273434 16617 raft_consensus.cc:482] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:51:43.273876 16617 raft_consensus.cc:504] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.276153 16617 leader_election.cc:288] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0d439ea757c48bf89bbd303dc6eaa0a (127.27.84.193:37397), e5111269974c411d9baaf2a72ca6d3b4 (127.27.84.194:38157)
I20260808 15:51:43.284561 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.276970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:50769 (local address 127.27.84.194:38157)
0808 15:51:43.277167 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.277173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.277644 (+ 471us) server_negotiation.cc:407] Connection header received
0808 15:51:43.277861 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.277867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.277944 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.278075 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:43.279054 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.279826 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.282457 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.283483 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.283722 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.283812 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.284151 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.284284 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:51:43.284921 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.276831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51749 (local address 127.27.84.193:37397)
0808 15:51:43.277127 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.277136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.277160 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:51:43.277248 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.277254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.277340 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.277446 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:43.277930 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.278947 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.281930 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.283513 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.283683 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.283768 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.284506 (+ 738us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.284657 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:51:43.284933 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.276418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.27.84.195:51749)
0808 15:51:43.276771 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.276792 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:43.276922 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:43.277493 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:43.277498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:43.277533 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:43.277756 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.277769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.279080 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.279085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.281474 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.281492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.283862 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.283869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.284056 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.284089 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:51:43.284249 (+ 160us) client_negotiation.cc:240] Negotiation successful
0808 15:51:43.284384 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":32}
I20260808 15:51:43.285620 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.276448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:38157 (local address 127.27.84.195:50769)
0808 15:51:43.277543 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.277561 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:43.277692 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:43.278081 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:43.278086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:43.278113 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:43.278828 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.278843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.279992 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.279997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.282290 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:43.282305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.283751 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.283758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:43.283982 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.284007 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:51:43.284122 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:51:43.284271 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":51,"thread_start_us":749,"threads_started":1}
I20260808 15:51:43.286294 16066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60826a8ab3ac4c409c4aa8b98372349e" candidate_uuid: "29ebf1ee19c245c582929a611fa5e311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" is_pre_election: true
I20260808 15:51:43.286826 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60826a8ab3ac4c409c4aa8b98372349e" candidate_uuid: "29ebf1ee19c245c582929a611fa5e311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5111269974c411d9baaf2a72ca6d3b4" is_pre_election: true
I20260808 15:51:43.287086 16066 raft_consensus.cc:2454] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29ebf1ee19c245c582929a611fa5e311 in term 0.
I20260808 15:51:43.287452 16286 raft_consensus.cc:2454] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29ebf1ee19c245c582929a611fa5e311 in term 0.
I20260808 15:51:43.287920 16397 leader_election.cc:302] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ebf1ee19c245c582929a611fa5e311, b0d439ea757c48bf89bbd303dc6eaa0a; no voters:
I20260808 15:51:43.288453 16617 raft_consensus.cc:2781] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:51:43.288640 16617 raft_consensus.cc:482] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:51:43.288789 16617 raft_consensus.cc:3037] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:43.292997 16617 raft_consensus.cc:504] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.293798 16617 leader_election.cc:288] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [CANDIDATE]: Term 1 election: Requested vote from peers b0d439ea757c48bf89bbd303dc6eaa0a (127.27.84.193:37397), e5111269974c411d9baaf2a72ca6d3b4 (127.27.84.194:38157)
I20260808 15:51:43.294350 16066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60826a8ab3ac4c409c4aa8b98372349e" candidate_uuid: "29ebf1ee19c245c582929a611fa5e311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a"
I20260808 15:51:43.294406 16286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60826a8ab3ac4c409c4aa8b98372349e" candidate_uuid: "29ebf1ee19c245c582929a611fa5e311" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5111269974c411d9baaf2a72ca6d3b4"
I20260808 15:51:43.294584 16066 raft_consensus.cc:3037] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:43.294652 16286 raft_consensus.cc:3037] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:43.297566 16066 raft_consensus.cc:2454] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ebf1ee19c245c582929a611fa5e311 in term 1.
I20260808 15:51:43.297991 16397 leader_election.cc:302] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ebf1ee19c245c582929a611fa5e311, b0d439ea757c48bf89bbd303dc6eaa0a; no voters:
I20260808 15:51:43.298420 16617 raft_consensus.cc:2781] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:51:43.298481 16286 raft_consensus.cc:2454] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29ebf1ee19c245c582929a611fa5e311 in term 1.
I20260808 15:51:43.299609 16617 raft_consensus.cc:686] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 1 LEADER]: Becoming Leader. State: Replica: 29ebf1ee19c245c582929a611fa5e311, State: Running, Role: LEADER
I20260808 15:51:43.300321 16617 consensus_queue.cc:237] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } }
I20260808 15:51:43.306447 15812 catalog_manager.cc:5495] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29ebf1ee19c245c582929a611fa5e311 (127.27.84.195). New cstate: current_term: 1 leader_uuid: "29ebf1ee19c245c582929a611fa5e311" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29ebf1ee19c245c582929a611fa5e311" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 35907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 } health_report { overall_health: UNKNOWN } } }
I20260808 15:51:43.333849 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.327130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.27.84.195:35907)
0808 15:51:43.327340 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.327345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.327365 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:51:43.327598 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.327602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.327696 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.327868 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:51:43.328436 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.329498 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.331319 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.331817 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.331960 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.332626 (+ 666us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:51:43.333173 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:51:43.333353 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.333405 (+ 52us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.333491 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:51:43.344543 16286 raft_consensus.cc:1261] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 1 FOLLOWER]: Refusing update from remote peer 29ebf1ee19c245c582929a611fa5e311: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:51:43.344648 16066 raft_consensus.cc:1261] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 1 FOLLOWER]: Refusing update from remote peer 29ebf1ee19c245c582929a611fa5e311: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:51:43.345700 16617 consensus_queue.cc:1035] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5111269974c411d9baaf2a72ca6d3b4" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:51:43.346038 16634 consensus_queue.cc:1035] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0d439ea757c48bf89bbd303dc6eaa0a" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:51:43.359724 16641 mvcc.cc:205] Tried to move back new op lower bound from 7316292826486861824 to 7316292826323103744. Current Snapshot: MvccSnapshot[applied={T|T < 7316292826486861824}]
I20260808 15:51:43.361933 16639 mvcc.cc:205] Tried to move back new op lower bound from 7316292826486861824 to 7316292826323103744. Current Snapshot: MvccSnapshot[applied={T|T < 7316292826486861824}]
I20260808 15:51:43.384651 15812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:51:43.385038 15812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:51:43.389411 15812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41cbfe5304d04ad9b7bae9a6fd7afd40: Sending DeleteTablet for 3 replicas of tablet 60826a8ab3ac4c409c4aa8b98372349e
I20260808 15:51:43.391016 16465 tablet_service.cc:1514] Processing DeleteTablet for tablet 60826a8ab3ac4c409c4aa8b98372349e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:51:43 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:38954
I20260808 15:51:43.391029 16035 tablet_service.cc:1514] Processing DeleteTablet for tablet 60826a8ab3ac4c409c4aa8b98372349e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:51:43 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:60230
I20260808 15:51:43.393002 16260 tablet_service.cc:1514] Processing DeleteTablet for tablet 60826a8ab3ac4c409c4aa8b98372349e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:51:43 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:42554
I20260808 15:51:43.399540 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 15874
I20260808 15:51:43.421615 16654 tablet_replica.cc:330] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: stopping tablet replica
I20260808 15:51:43.422317 16654 raft_consensus.cc:2229] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:51:43.423105 16653 tablet_replica.cc:330] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: stopping tablet replica
I20260808 15:51:43.423149 16654 raft_consensus.cc:2258] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:51:43.425872 16654 ts_tablet_manager.cc:1905] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:51:43.423580 16653 raft_consensus.cc:2229] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:51:43.428237 16653 raft_consensus.cc:2258] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:51:43.429390 16650 tablet_replica.cc:330] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: stopping tablet replica
I20260808 15:51:43.430313 16650 raft_consensus.cc:2229] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:51:43.431258 16650 raft_consensus.cc:2258] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:51:43.434511 16653 ts_tablet_manager.cc:1905] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:51:43.436643 16654 ts_tablet_manager.cc:1918] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:51:43.436944 16654 log.cc:1196] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/60826a8ab3ac4c409c4aa8b98372349e
I20260808 15:51:43.437762 16654 ts_tablet_manager.cc:1939] T 60826a8ab3ac4c409c4aa8b98372349e P e5111269974c411d9baaf2a72ca6d3b4: Deleting consensus metadata
I20260808 15:51:43.439932 15777 catalog_manager.cc:4841] TS e5111269974c411d9baaf2a72ca6d3b4 (127.27.84.194:38157): tablet 60826a8ab3ac4c409c4aa8b98372349e (table test-table [id=963e895f9fe94c5cb6bf84aef610f4d2]) successfully deleted
I20260808 15:51:43.441159 16650 ts_tablet_manager.cc:1905] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:51:43.450598 16650 ts_tablet_manager.cc:1918] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:51:43.451041 16650 log.cc:1196] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/60826a8ab3ac4c409c4aa8b98372349e
I20260808 15:51:43.451848 16650 ts_tablet_manager.cc:1939] T 60826a8ab3ac4c409c4aa8b98372349e P 29ebf1ee19c245c582929a611fa5e311: Deleting consensus metadata
I20260808 15:51:43.454120 15779 catalog_manager.cc:4841] TS 29ebf1ee19c245c582929a611fa5e311 (127.27.84.195:35907): tablet 60826a8ab3ac4c409c4aa8b98372349e (table test-table [id=963e895f9fe94c5cb6bf84aef610f4d2]) successfully deleted
I20260808 15:51:43.455657 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.401850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60236 (local address 127.27.84.193:37397)
0808 15:51:43.402043 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.402050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.419374 (+ 17324us) server_negotiation.cc:407] Connection header received
0808 15:51:43.419976 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.419983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.420080 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.420165 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:43.424111 (+ 3946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.425333 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.431728 (+ 6395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.432433 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.432546 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.442894 (+ 10348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:43.442920 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:43.442938 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:43.442998 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:43.448095 (+ 5097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:43.450955 (+ 2860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:43.450965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:43.450969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:43.451206 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:43.452409 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:43.452414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:43.452417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:43.452921 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:43.453219 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.454424 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.454913 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260808 15:51:43.456339 16126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:43.456622 16653 ts_tablet_manager.cc:1918] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:51:43.456957 16653 log.cc:1196] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/60826a8ab3ac4c409c4aa8b98372349e
I20260808 15:51:43.567555 16653 ts_tablet_manager.cc:1939] T 60826a8ab3ac4c409c4aa8b98372349e P b0d439ea757c48bf89bbd303dc6eaa0a: Deleting consensus metadata
I20260808 15:51:43.567595 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 15874
W20260808 15:51:43.593400 15780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:37397 (error 108)
I20260808 15:51:43.593731 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 16138
I20260808 15:51:43.607224 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.606682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60240)
0808 15:51:43.606889 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.607031 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 15:51:43.607651 15780 catalog_manager.cc:4584] TS b0d439ea757c48bf89bbd303dc6eaa0a (127.27.84.193:37397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60826a8ab3ac4c409c4aa8b98372349e: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
I20260808 15:51:43.615084 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.595012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.27.84.194:38157)
0808 15:51:43.595281 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.595290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.596180 (+ 890us) server_negotiation.cc:407] Connection header received
0808 15:51:43.596293 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.596302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.596412 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.596527 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:43.599336 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.600838 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.603532 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.604435 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.604589 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.608638 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:43.608652 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:43.608667 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:43.608707 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:43.611902 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:43.612643 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:43.612650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:43.612652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:43.612835 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:43.613264 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:43.613270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:43.613272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:43.613652 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:43.613987 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.614291 (+ 304us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.614551 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:51:43.615988 16347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:43.661806 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.661320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60256)
0808 15:51:43.661510 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.661617 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:51:43.663025 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.662663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60264)
0808 15:51:43.662799 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.662875 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260808 15:51:43.710453 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 16138
I20260808 15:51:43.723557 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.723073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60278)
0808 15:51:43.723297 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.723385 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:51:43.724699 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.724365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60288)
0808 15:51:43.724520 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.724596 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:51:43.744335 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 16362
I20260808 15:51:43.761518 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.745294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.27.84.195:35907)
0808 15:51:43.745509 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:43.745516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:43.745540 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:51:43.745769 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:43.745776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:43.745872 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:43.746001 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:43.747610 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.748860 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.750320 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:43.751046 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:43.751184 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:43.754082 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:43.754096 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:43.754115 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:43.754170 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:43.758321 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:43.759188 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:43.759197 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:43.759201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:43.759417 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:43.759779 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:43.759785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:43.759789 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:43.760277 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:43.760597 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:43.760844 (+ 247us) server_negotiation.cc:299] Negotiation successful
0808 15:51:43.761073 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:51:43.762356 16563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:43.805497 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.804970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60294)
0808 15:51:43.805207 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.805305 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260808 15:51:43.806674 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:43.806222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60296)
0808 15:51:43.806441 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:43.806515 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260808 15:51:43.883538 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 16362
I20260808 15:51:43.910319 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 15741
I20260808 15:51:43.911655 15865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:44.031975 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:44.031379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37397 (local address 127.0.0.1:60308)
0808 15:51:44.031643 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:44.031775 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:51:44.032289 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 15741
I20260808 15:51:44.066700 27987 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260808 15:51:44.412355 27987 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260808 15:51:44.412633 27987 mini_ranger.cc:66] Stopping Ranger...
I20260808 15:51:44.758597 27987 mini_ranger.cc:68] Stopped Ranger
2026-08-08 15:51:44.758 UTC [14239] LOG: received smart shutdown request
2026-08-08 15:51:44.762 UTC [14239] LOG: background worker "logical replication launcher" (PID 14254) exited with exit code 1
2026-08-08 15:51:44.762 UTC [14244] LOG: shutting down
2026-08-08 15:51:44.943 UTC [14239] LOG: database system is shut down
2026-08-08T15:51:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54975 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:51:45 dist-test-slave-twwt krb5kdc[16677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:51:45 dist-test-slave-twwt krb5kdc[16677](info): set up 2 sockets
Aug 08 15:51:45 dist-test-slave-twwt krb5kdc[16677](info): commencing operation
krb5kdc: starting...
W20260808 15:51:46.435360 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.310s user 0.000s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:51:46 dist-test-slave-twwt krb5kdc[16677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:51:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:51:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:51:46.738227 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36031
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36031,127.27.84.253:41443,127.27.84.252:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:47.089628 16696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:47.089912 16696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:47.090087 16696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:47.097389 16696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:47.097515 16696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:47.097602 16696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:47.097699 16696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:47.097782 16696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:47.107632 16696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36031,127.27.84.253:41443,127.27.84.252:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:47.108946 16696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:47.110304 16696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:47.122180 16705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:47.122335 16706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:47.122985 16709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:47.123387 16696 server_base.cc:1034] running on GCE node
I20260808 15:51:47.124292 16696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:47.125802 16696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:47.127076 16696 hybrid_clock.cc:648] HybridClock initialized: now 1786204307127012 us; error 63 us; skew 500 ppm
I20260808 15:51:47.127458 16696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:47 dist-test-slave-twwt krb5kdc[16677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:47.130817 16696 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:51:47.132293 16696 webserver.cc:460] Webserver started at http://127.27.84.254:42093/ using document root <none> and password file <none>
I20260808 15:51:47.132982 16696 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:47.133082 16696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:47.133373 16696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:47.136435 16696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30"
format_stamp: "Formatted at 2026-08-08 15:51:47 on dist-test-slave-twwt"
server_key: "bfc90c0225ab15d5cb328e9fe9da0868"
server_key_iv: "99eb81476a33fee0198182e11c815ceb"
server_key_version: "encryptionkey@0"
I20260808 15:51:47.137285 16696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30"
format_stamp: "Formatted at 2026-08-08 15:51:47 on dist-test-slave-twwt"
server_key: "bfc90c0225ab15d5cb328e9fe9da0868"
server_key_iv: "99eb81476a33fee0198182e11c815ceb"
server_key_version: "encryptionkey@0"
I20260808 15:51:47.142875 16696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:51:47.146781 16717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:47.148389 16696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:51:47.148604 16696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30"
format_stamp: "Formatted at 2026-08-08 15:51:47 on dist-test-slave-twwt"
server_key: "bfc90c0225ab15d5cb328e9fe9da0868"
server_key_iv: "99eb81476a33fee0198182e11c815ceb"
server_key_version: "encryptionkey@0"
I20260808 15:51:47.148838 16696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:47.167032 16696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:47.171537 16696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:47.171849 16696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:47.191968 16696 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:36031
I20260808 15:51:47.192031 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:36031 every 8 connection(s)
I20260808 15:51:47.193442 16696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:51:47.197399 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:47.198975 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 16696
I20260808 15:51:47.199232 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:51:47.200106 27987 external_mini_cluster.cc:1428] Setting key 95e326280f813fffe118a4b5c3f02242
I20260808 15:51:47.207547 16808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:51:47 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.84.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.253@KRBTEST.COM" created.
I20260808 15:51:47.298817 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.223578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.27.84.254:36031)
0808 15:51:47.227693 (+ 4115us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:47.227716 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:47.227824 (+ 108us) server_negotiation.cc:407] Connection header received
0808 15:51:47.230539 (+ 2715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:47.230569 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:47.232771 (+ 2202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:47.233802 (+ 1031us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:47.238178 (+ 4376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:47.247462 (+ 9284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:47.250530 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:47.252880 (+ 2350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:47.253365 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:47.264155 (+ 10790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:47.264277 (+ 122us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:47.264355 (+ 78us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:47.264638 (+ 283us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:47.286542 (+ 21904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:47.288195 (+ 1653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:47.288206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:47.288208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:47.288603 (+ 395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:47.289564 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:47.289584 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:47.289590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:47.290946 (+ 1356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:47.291259 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:47.293877 (+ 2618us) server_negotiation.cc:299] Negotiation successful
0808 15:51:47.294835 (+ 958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":219,"server-negotiator.queue_time_us":3606,"spinlock_wait_cycles":112000,"thread_start_us":3140,"threads_started":1}
I20260808 15:51:47.298877 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.223578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36031 (local address 127.0.0.1:35416)
0808 15:51:47.225793 (+ 2215us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.225900 (+ 107us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:47.229738 (+ 3838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:47.233652 (+ 3914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:47.233667 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:47.235800 (+ 2133us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:47.237397 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:47.237479 (+ 82us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:47.247857 (+ 10378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:47.247862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:47.250185 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:47.250221 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:47.253541 (+ 3320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:47.253565 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:47.254043 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:47.255775 (+ 1732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:47.255838 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:47.262913 (+ 7075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:47.286966 (+ 24053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:47.286989 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:47.287009 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:47.287944 (+ 935us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:47.288796 (+ 852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:47.288800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:47.288802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:47.289277 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:47.291337 (+ 2060us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:47.291353 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:47.292445 (+ 1092us) client_negotiation.cc:769] Sending connection context
0808 15:51:47.293680 (+ 1235us) client_negotiation.cc:240] Negotiation successful
0808 15:51:47.295840 (+ 2160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"mutex_wait_us":51,"thread_start_us":880,"threads_started":1}
I20260808 15:51:47.309985 16808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:51:47.312759 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.312032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36756)
0808 15:51:47.312293 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.312560 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260808 15:51:47.313380 16729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
I20260808 15:51:47.320040 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.318759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36764)
0808 15:51:47.319271 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.319548 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260808 15:51:47.320704 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41443: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
I20260808 15:51:47.370841 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } attempt: 1
I20260808 15:51:47.375092 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.373496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36778)
0808 15:51:47.374130 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.374524 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285}
I20260808 15:51:47.377506 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.376722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36792)
0808 15:51:47.376962 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.377089 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260808 15:51:47.377738 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41443: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab.
Entry for principal kudu/127.27.84.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab.
W20260808 15:51:47.388341 27987 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.84.253: real 0.188s user 0.007s sys 0.059s
I20260808 15:51:47.390094 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:41443
--webserver_interface=127.27.84.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab
--principal=kudu/127.27.84.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36031,127.27.84.253:41443,127.27.84.252:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:51:47.446198 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } attempt: 2
I20260808 15:51:47.450549 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.449145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36808)
0808 15:51:47.449794 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.450062 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260808 15:51:47.454967 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.452709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36816)
0808 15:51:47.453615 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.454156 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260808 15:51:47.455492 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41443: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
I20260808 15:51:47.546896 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } attempt: 3
I20260808 15:51:47.548898 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.548187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36826)
0808 15:51:47.548514 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.548692 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260808 15:51:47.550817 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.550058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36830)
0808 15:51:47.550335 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.550460 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260808 15:51:47.551151 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41443: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
I20260808 15:51:47.694446 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } attempt: 4
I20260808 15:51:47.695928 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.695248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36846)
0808 15:51:47.695622 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.695741 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202}
I20260808 15:51:47.697089 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.696657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36852)
0808 15:51:47.696819 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.696932 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 15:51:47.697311 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41443: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
W20260808 15:51:47.946025 16817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:47.946333 16817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:47.946529 16817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:47.956655 16817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:47.956848 16817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:47.956935 16817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:47.957022 16817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:47.957093 16817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:47.967672 16817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36031,127.27.84.253:41443,127.27.84.252:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab
--principal=kudu/127.27.84.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.84.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:47.969043 16817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:47.970429 16817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:47.982959 16833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:47.983196 16835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:47.983044 16831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:47.984037 16817 server_base.cc:1034] running on GCE node
I20260808 15:51:47.984853 16817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:47.986263 16817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:47.987497 16817 hybrid_clock.cc:648] HybridClock initialized: now 1786204307987457 us; error 55 us; skew 500 ppm
I20260808 15:51:47.987865 16817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:47 dist-test-slave-twwt krb5kdc[16677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:47.991293 16817 init.cc:377] Logged in from keytab as kudu/127.27.84.253@KRBTEST.COM (short username kudu)
I20260808 15:51:47.992798 16817 webserver.cc:460] Webserver started at http://127.27.84.253:37091/ using document root <none> and password file <none>
I20260808 15:51:47.993471 16817 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:47.993577 16817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:47.993831 16817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:47.996560 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } attempt: 5
I20260808 15:51:47.996949 16817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data/instance:
uuid: "593837f757474ef4993034d411803eb4"
format_stamp: "Formatted at 2026-08-08 15:51:47 on dist-test-slave-twwt"
server_key: "b45e99e5ef139e49f08a4fe5153080eb"
server_key_iv: "4251eee5b40d225030abc9fbc7b5fa62"
server_key_version: "encryptionkey@0"
I20260808 15:51:47.997669 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.997207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36868)
0808 15:51:47.997395 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.997524 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 15:51:47.998052 16817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal/instance:
uuid: "593837f757474ef4993034d411803eb4"
format_stamp: "Formatted at 2026-08-08 15:51:47 on dist-test-slave-twwt"
server_key: "b45e99e5ef139e49f08a4fe5153080eb"
server_key_iv: "4251eee5b40d225030abc9fbc7b5fa62"
server_key_version: "encryptionkey@0"
I20260808 15:51:47.998812 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:47.998448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36870)
0808 15:51:47.998582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:47.998693 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 15:51:47.999069 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41443: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41443: connect: Connection refused (error 111)
I20260808 15:51:48.003638 16817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:51:48.007181 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:48.008539 16817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:51:48.008721 16817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal
uuid: "593837f757474ef4993034d411803eb4"
format_stamp: "Formatted at 2026-08-08 15:51:47 on dist-test-slave-twwt"
server_key: "b45e99e5ef139e49f08a4fe5153080eb"
server_key_iv: "4251eee5b40d225030abc9fbc7b5fa62"
server_key_version: "encryptionkey@0"
I20260808 15:51:48.008936 16817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:48.048389 16817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:48.053413 16817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:48.053748 16817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:48.073721 16817 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.253:41443
I20260808 15:51:48.073758 16919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.253:41443 every 8 connection(s)
I20260808 15:51:48.075296 16817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
I20260808 15:51:48.076560 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 16817
I20260808 15:51:48.076778 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-1/wal/instance
I20260808 15:51:48.077358 27987 external_mini_cluster.cc:1428] Setting key 9e74b3cfc539b463daa065cf3f1aaac1
I20260808 15:51:48.080293 16923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:48.090469 16923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.84.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.252@KRBTEST.COM" created.
I20260808 15:51:48.112311 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.093272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.27.84.254:36031)
0808 15:51:48.093843 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.093848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.094984 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 15:51:48.096951 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.096957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.097049 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.097177 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.098827 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.099657 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.101250 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.101687 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.101828 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.105693 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.105698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.105701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.105735 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.108579 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.109459 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.109463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.109465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.109592 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.110145 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.110152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.110157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.110583 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.110764 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.111479 (+ 715us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.111768 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":331,"threads_started":1}
I20260808 15:51:48.113006 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.093904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36031 (local address 127.0.0.1:35420)
0808 15:51:48.094802 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.094850 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.096647 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.097454 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.097464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.098005 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.098596 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.098671 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.099839 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.099844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.101093 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.101106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.101933 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.101941 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.102173 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.102934 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.102962 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.105456 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.108818 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.108828 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.108846 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.109308 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.109738 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.109745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.109752 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.110013 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.110726 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.110732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.111078 (+ 346us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.111330 (+ 252us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.111737 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":342,"threads_started":1}
I20260808 15:51:48.115072 16923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:51:48.130800 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.115931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36878)
0808 15:51:48.116165 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.116188 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.116304 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.117341 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.117352 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.117858 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.118079 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.118093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.119537 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.119545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.121063 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.121081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.121922 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.121927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.122137 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.122738 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.122755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.124828 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.128145 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.128153 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.128160 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.128547 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.128949 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.128955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.128959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.129105 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.129711 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.129722 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.129862 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.130014 (+ 152us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.130311 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260808 15:51:48.130892 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.116177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36878 (local address 127.27.84.253:41443)
0808 15:51:48.116913 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.116919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.116946 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:51:48.116989 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.116994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.117168 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.117289 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.118239 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.119390 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.121210 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.121744 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.121898 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.125062 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.125072 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.125086 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.125113 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.128004 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.128687 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.128700 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.128702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.128837 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.129213 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.129220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.129223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.129584 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.129803 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.130151 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.130439 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":580,"thread_start_us":425,"threads_started":1}
I20260808 15:51:48.133266 16923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:51:48.134652 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.134133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33672)
0808 15:51:48.134318 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.134462 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 15:51:48.134939 16848 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
I20260808 15:51:48.136783 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.136439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33678)
0808 15:51:48.136592 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.136679 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:51:48.137041 16923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:40991: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab.
Entry for principal kudu/127.27.84.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab.
I20260808 15:51:48.146219 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:40991
--webserver_interface=127.27.84.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab
--principal=kudu/127.27.84.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:41199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36031,127.27.84.253:41443,127.27.84.252:40991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:51:48.186342 16923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } attempt: 1
I20260808 15:51:48.187649 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.187145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33688)
0808 15:51:48.187379 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.187492 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260808 15:51:48.188740 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33694)
0808 15:51:48.188544 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.188618 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260808 15:51:48.189029 16923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:40991: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
I20260808 15:51:48.257251 16923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } attempt: 2
I20260808 15:51:48.258526 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.258047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33706)
0808 15:51:48.258251 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.258361 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 15:51:48.259595 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.259235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33720)
0808 15:51:48.259409 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.259481 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 15:51:48.259810 16923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:40991: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
I20260808 15:51:48.351032 16923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } attempt: 3
I20260808 15:51:48.352245 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.351751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33722)
0808 15:51:48.351984 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.352085 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260808 15:51:48.353490 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.352980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33738)
0808 15:51:48.353236 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.353363 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260808 15:51:48.353677 16923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:40991: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
I20260808 15:51:48.496881 16923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } attempt: 4
I20260808 15:51:48.498099 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.497606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33750)
0808 15:51:48.497824 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.497926 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260808 15:51:48.499243 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.498889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33752)
0808 15:51:48.499062 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.499136 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260808 15:51:48.499529 16923 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:40991: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
W20260808 15:51:48.503350 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:48.503626 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:48.503726 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:48.511215 16932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:48.511319 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:48.511365 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:48.511416 16932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:48.511462 16932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:48.521592 16932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36031,127.27.84.253:41443,127.27.84.252:40991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab
--principal=kudu/127.27.84.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:40991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.84.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:48.522935 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:48.524230 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:48.536466 16941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:48.536456 16938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:48.536906 16939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:48.537354 16932 server_base.cc:1034] running on GCE node
I20260808 15:51:48.538117 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:48.539549 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:48.540799 16932 hybrid_clock.cc:648] HybridClock initialized: now 1786204308540723 us; error 35 us; skew 500 ppm
I20260808 15:51:48.541186 16932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204308, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:48.544518 16932 init.cc:377] Logged in from keytab as kudu/127.27.84.252@KRBTEST.COM (short username kudu)
I20260808 15:51:48.546105 16932 webserver.cc:460] Webserver started at http://127.27.84.252:46107/ using document root <none> and password file <none>
I20260808 15:51:48.546298 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } attempt: 6
I20260808 15:51:48.547066 16932 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:48.547222 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:48.547588 16932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:48.550835 16932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data/instance:
uuid: "6465fc4f712341059594d57095bbdcb7"
format_stamp: "Formatted at 2026-08-08 15:51:48 on dist-test-slave-twwt"
server_key: "c477df06f4aa86b6b213021e6726d90f"
server_key_iv: "31ac2ff0c09dfcd9a6eb94c91215665e"
server_key_version: "encryptionkey@0"
I20260808 15:51:48.551695 16932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal/instance:
uuid: "6465fc4f712341059594d57095bbdcb7"
format_stamp: "Formatted at 2026-08-08 15:51:48 on dist-test-slave-twwt"
server_key: "c477df06f4aa86b6b213021e6726d90f"
server_key_iv: "31ac2ff0c09dfcd9a6eb94c91215665e"
server_key_version: "encryptionkey@0"
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:51:48.557542 16932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:51:48.560361 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.547066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36882 (local address 127.27.84.253:41443)
0808 15:51:48.547258 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.547267 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.547617 (+ 350us) server_negotiation.cc:407] Connection header received
0808 15:51:48.547807 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.547811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.547878 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.547994 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.548744 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.549591 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.550885 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.551440 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.551577 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.554901 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.554907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.554910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.554942 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.557677 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.558387 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.558394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.558398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.558532 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.558936 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.558941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.558945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.559236 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.559411 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.559866 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.560113 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:51:48.560434 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.546986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36882)
0808 15:51:48.547504 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.547519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.547675 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.548031 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.548036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.548405 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.548569 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.548578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.549748 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.549755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.550743 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.550756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.551593 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.551599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.551787 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.552430 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.552450 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.554702 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.557882 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.557887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.557891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.558268 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.558669 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.558674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.558677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.558820 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.559366 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.559371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.559525 (+ 154us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.559693 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.559958 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":301,"threads_started":1}
I20260808 15:51:48.561682 16808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:51:48.561830 16955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:48.562951 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.562433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33760)
0808 15:51:48.562697 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.562798 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 15:51:48.563524 16932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:51:48.563726 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal
uuid: "6465fc4f712341059594d57095bbdcb7"
format_stamp: "Formatted at 2026-08-08 15:51:48 on dist-test-slave-twwt"
server_key: "c477df06f4aa86b6b213021e6726d90f"
server_key_iv: "31ac2ff0c09dfcd9a6eb94c91215665e"
server_key_version: "encryptionkey@0"
I20260808 15:51:48.563951 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:48.564505 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.564085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33768)
0808 15:51:48.564245 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.564340 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 15:51:48.564793 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:40991: Network error: Client connection negotiation failed: client connection to 127.27.84.252:40991: connect: Connection refused (error 111)
I20260808 15:51:48.587419 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:48.592075 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:48.592391 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:48.610584 16932 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.252:40991
I20260808 15:51:48.610667 17031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.252:40991 every 8 connection(s)
I20260808 15:51:48.612040 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
I20260808 15:51:48.616147 17032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:48.616696 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 16932
I20260808 15:51:48.616906 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786204089698720-27987-0/minicluster-data/master-2/wal/instance
I20260808 15:51:48.617033 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } attempt: 1
I20260808 15:51:48.617547 27987 external_mini_cluster.cc:1428] Setting key ee5df52cde80ac9c983928344d0cf325
I20260808 15:51:48.625967 17032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:51:48.643968 16951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.617718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33780)
0808 15:51:48.618020 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.618037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.618138 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.624078 (+ 5940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.624085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.624597 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.624796 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.624809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.627131 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.627138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.628528 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.628548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.630261 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.630271 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.630541 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.631445 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.631479 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.635446 (+ 3967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.640626 (+ 5180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.640632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.640638 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.641128 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.641651 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.641656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.641660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.641809 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.642873 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.642878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.643102 (+ 224us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.643272 (+ 170us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.643583 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260808 15:51:48.645965 17038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.619092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.27.84.252:40991)
0808 15:51:48.620470 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.620487 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.620538 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:51:48.623056 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.623089 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.623639 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.624092 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.624972 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.626956 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.628692 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.630012 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.631106 (+ 1094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.636058 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.636074 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.636104 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.636171 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.640462 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.641281 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.641296 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.641303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.641512 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.641940 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.641947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.641954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.642532 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.642795 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.643602 (+ 807us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.644405 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":503,"server-negotiator.queue_time_us":1101,"spinlock_wait_cycles":9984,"thread_start_us":992,"threads_started":1}
I20260808 15:51:48.645994 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.619095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.27.84.252:40991)
0808 15:51:48.620470 (+ 1375us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.620487 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.620538 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:51:48.623056 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.623089 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.623639 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.624092 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.626135 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.627404 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.629440 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.630334 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.638750 (+ 8416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.639000 (+ 250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.639004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.639008 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.639028 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.642702 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.643469 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.643474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.643477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.643617 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.644142 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.644149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.644154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.644527 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.644771 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.645202 (+ 431us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.645402 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":372,"server-negotiator.queue_time_us":1188,"spinlock_wait_cycles":32896,"thread_start_us":531,"threads_started":1}
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204308, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:48.654649 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30: Bootstrap starting.
I20260808 15:51:48.658598 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.629006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:36031 (local address 127.0.0.1:35432)
0808 15:51:48.638683 (+ 9677us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.638756 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.638935 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.640953 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.640961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.642044 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.642494 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.642516 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.644167 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.644171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.646020 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.646034 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.647057 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.647064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.647232 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.647920 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.647969 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.650890 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.655431 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.655443 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.655450 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.655973 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.656452 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.656459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.656464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.656661 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.657444 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.657455 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.657645 (+ 190us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.657832 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.658184 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9324,"thread_start_us":8328,"threads_started":1}
I20260808 15:51:48.659189 16808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:48.659977 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.628946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.27.84.254:36031)
0808 15:51:48.640430 (+ 11484us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.640435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.640456 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:51:48.640514 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.640522 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.640609 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.640710 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.642713 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.643928 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.646181 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.646846 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.646978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.651152 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.651159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.651164 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.651211 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.655251 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.656145 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.656150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.656153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.656295 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.656824 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.656828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.656832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.657274 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.659051 (+ 1777us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.659218 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.659475 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11350,"thread_start_us":9675,"threads_started":1}
I20260808 15:51:48.660852 16808 log.cc:824] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:48.661285 17032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:51:48.664496 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30: No bootstrap required, opened a new log
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204308, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:51:48.674727 16808 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.675125 16808 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:48.675274 16808 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cdfc3baaf04bb6a6209c0cb8004f30, State: Initialized, Role: FOLLOWER
I20260808 15:51:48.675988 16808 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.677019 17044 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.677487 17044 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.678282 16808 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:48.678314 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.662064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41443 (local address 127.0.0.1:36894)
0808 15:51:48.662500 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.662520 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.662647 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.663369 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.663376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.663845 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.664059 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.664073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.665443 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.665448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.666828 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.666845 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.667941 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.667948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.668146 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.668790 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.668812 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.671538 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.675330 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.675334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.675336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.675985 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.676454 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.676459 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.676465 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.676641 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.677355 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.677360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.677523 (+ 163us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.677670 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.677942 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260808 15:51:48.678737 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.649877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.27.84.254:36031)
0808 15:51:48.659199 (+ 9322us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.659204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.659227 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:51:48.659270 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.659274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.659337 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.659513 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.661349 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.662443 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.664088 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.664773 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.665605 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.670023 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.670029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.670035 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.670065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.674233 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.675178 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.675184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.675190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.675373 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.675900 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.675908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.675912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.676288 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.676609 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.677885 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.678103 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9183,"thread_start_us":8953,"threads_started":1}
I20260808 15:51:48.679175 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.662779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.27.84.253:41443)
0808 15:51:48.663031 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.663038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.663058 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:51:48.663118 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.663124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.663198 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.663306 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.664206 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.665258 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.667006 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.667749 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.667879 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.671774 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.671781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.671785 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.671830 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.675133 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.676132 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.676137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.676140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.676304 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.676778 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.676783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.676786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.677130 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.677269 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.678647 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.678897 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:51:48.680210 17032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204308, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:51:48.697779 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.681312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.27.84.252:40991)
0808 15:51:48.681571 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.681576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.682023 (+ 447us) server_negotiation.cc:407] Connection header received
0808 15:51:48.682149 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.682158 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.682254 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.682381 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.683348 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.684414 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.686093 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.686971 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.687119 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.691171 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.691178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.691182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.691216 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.694778 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.695564 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.695569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.695573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.695709 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.696129 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.696134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.696137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.696555 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.696676 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.697183 (+ 507us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.697415 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260808 15:51:48.698937 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.681100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33800)
0808 15:51:48.681909 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.681928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.682028 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.682424 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.682431 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.682959 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.683187 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.683201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.684567 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.684571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.685920 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.685935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.687325 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.687334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.687505 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.688172 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.688202 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.690988 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.694939 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.694943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.694947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.695439 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.695857 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.695861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.695864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.696021 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.696759 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.696766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.696918 (+ 152us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.698218 (+ 1300us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.698551 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700}
I20260808 15:51:48.702458 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.681729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.27.84.253:41443)
0808 15:51:48.681876 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.681882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.681898 (+ 16us) server_negotiation.cc:407] Connection header received
0808 15:51:48.681948 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.681956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.682027 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.682123 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.683964 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.685014 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.686639 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.687380 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.687515 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.693905 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.693914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.693922 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.693972 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.698456 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.699880 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.699886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.699889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.700368 (+ 479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.700868 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.700875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.700878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.701246 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.701524 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.701870 (+ 346us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.702114 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260808 15:51:48.693238 16696 master_runner.cc:186] Error getting master registration for 127.27.84.253:41443: OK, instance_id { permanent_uuid: "593837f757474ef4993034d411803eb4" instance_seqno: 1786204308068013 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:51:48.703799 16696 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:51:48.705178 17032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: Bootstrap starting.
I20260808 15:51:48.707525 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260808 15:51:48.707773 17059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:51:48.707904 17059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:51:48.709070 17032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:48.710144 17032 log.cc:824] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:48.713336 17032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: No bootstrap required, opened a new log
I20260808 15:51:48.733078 17032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.733713 17032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:48.733853 17032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6465fc4f712341059594d57095bbdcb7, State: Initialized, Role: FOLLOWER
I20260808 15:51:48.736758 17032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.753094 17042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.716710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.27.84.254:36031)
0808 15:51:48.716926 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.716932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.727459 (+ 10527us) server_negotiation.cc:407] Connection header received
0808 15:51:48.727828 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.727838 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.727987 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.728085 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.736314 (+ 8229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.737444 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.740260 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.740982 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.741092 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.745096 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.745103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.745108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.745154 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.749116 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.750110 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.750117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.750120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.750292 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.750827 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.750832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.750836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.751241 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.751521 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.752282 (+ 761us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.752611 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260808 15:51:48.758348 17074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.759222 17074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.760480 17032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:48.765367 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.716333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.27.84.252:40991)
0808 15:51:48.716558 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.716565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.735087 (+ 18522us) server_negotiation.cc:407] Connection header received
0808 15:51:48.735469 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.735478 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.735571 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.735653 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.741819 (+ 6166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.743314 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.750280 (+ 6966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.751105 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.751222 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.757785 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.757793 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.757798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.757848 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.761779 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.762722 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.762728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.762731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.762936 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.763427 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.763433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.763436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.763841 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.764108 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.764727 (+ 619us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.764982 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:51:48.771656 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.722567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.27.84.253:41443)
0808 15:51:48.722797 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.722804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.737698 (+ 14894us) server_negotiation.cc:407] Connection header received
0808 15:51:48.737952 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.737959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.738045 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.738163 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.745964 (+ 7801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.747457 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.753872 (+ 6415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.754559 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.754702 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.762055 (+ 7353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.762066 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.762074 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.762135 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.766840 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.768683 (+ 1843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.768693 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.768699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.768929 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.769479 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.769487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.769493 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.769983 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.770306 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.770748 (+ 442us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.771102 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260808 15:51:48.777103 17044 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:51:48.777424 17044 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.779162 17044 leader_election.cc:288] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593837f757474ef4993034d411803eb4 (127.27.84.253:41443), 6465fc4f712341059594d57095bbdcb7 (127.27.84.252:40991)
I20260808 15:51:48.780239 16998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6465fc4f712341059594d57095bbdcb7" is_pre_election: true
I20260808 15:51:48.780934 16998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5cdfc3baaf04bb6a6209c0cb8004f30 in term 0.
I20260808 15:51:48.781574 16725 leader_election.cc:302] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6465fc4f712341059594d57095bbdcb7, b5cdfc3baaf04bb6a6209c0cb8004f30; no voters:
I20260808 15:51:48.781986 17044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:51:48.782146 17044 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:51:48.782254 17044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:48.785517 17044 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.786809 17044 leader_election.cc:288] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [CANDIDATE]: Term 1 election: Requested vote from peers 593837f757474ef4993034d411803eb4 (127.27.84.253:41443), 6465fc4f712341059594d57095bbdcb7 (127.27.84.252:40991)
I20260808 15:51:48.786791 16998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6465fc4f712341059594d57095bbdcb7"
I20260808 15:51:48.787025 16998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:48.790656 16998 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5cdfc3baaf04bb6a6209c0cb8004f30 in term 1.
I20260808 15:51:48.791332 16725 leader_election.cc:302] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6465fc4f712341059594d57095bbdcb7, b5cdfc3baaf04bb6a6209c0cb8004f30; no voters:
I20260808 15:51:48.791744 17044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 1 FOLLOWER]: Leader election won for term 1
W20260808 15:51:48.797269 17092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:51:48.797412 17092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:51:48.797809 17044 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [term 1 LEADER]: Becoming Leader. State: Replica: b5cdfc3baaf04bb6a6209c0cb8004f30, State: Running, Role: LEADER
I20260808 15:51:48.798746 16923 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } attempt: 5
I20260808 15:51:48.799396 17044 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
Aug 08 15:51:48 dist-test-slave-twwt krb5kdc[16677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:51:48.818161 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.799316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:40991 (local address 127.0.0.1:33822)
0808 15:51:48.799655 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:48.799680 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:48.799798 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:48.800658 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:48.800664 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:48.801250 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:48.801477 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.801490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.803132 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.803139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.804575 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:48.804593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.806122 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.806129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:48.806385 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.807126 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:48.807152 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:48.810385 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:48.814835 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.814842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.814849 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.815421 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.815904 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:48.815909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:48.815913 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:48.816097 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:48.817032 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:48.817039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:48.817212 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 15:51:48.817387 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 15:51:48.817715 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260808 15:51:48.819264 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:48.800016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.27.84.252:40991)
0808 15:51:48.800185 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:48.800190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:48.800208 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:51:48.800260 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:48.800264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:48.800357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:48.800458 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:48.801690 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.802928 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.804781 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:48.805482 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:48.805615 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:48.810640 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:48.810647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:48.810652 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:48.810695 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:48.814635 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.815570 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.815576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.815580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.815760 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:48.816238 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:48.816243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:48.816246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:48.816801 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:48.816963 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:48.818650 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0808 15:51:48.818908 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:51:48.820052 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdfc3baaf04bb6a6209c0cb8004f30. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.820538 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:48.827662 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: Bootstrap starting.
I20260808 15:51:48.831187 17099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:51:48.833452 17099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:51:48.796044 16932 master_runner.cc:186] Error getting master registration for 127.27.84.253:41443: OK, instance_id { permanent_uuid: "593837f757474ef4993034d411803eb4" instance_seqno: 1786204308068013 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:51:48.841288 16932 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:51:48.842341 16923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:48.843842 16923 log.cc:824] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: Log is configured to *not* fsync() on all Append() calls
W20260808 15:51:48.844714 16966 tablet.cc:2369] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:51:48.845542 16998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [term 1 FOLLOWER]: Refusing update from remote peer b5cdfc3baaf04bb6a6209c0cb8004f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:51:48.846623 17094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:51:48.855931 16923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: No bootstrap required, opened a new log
I20260808 15:51:48.877751 16923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.878249 16923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:48.882977 17074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdfc3baaf04bb6a6209c0cb8004f30. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.883327 17074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.884857 17100 mvcc.cc:205] Tried to move back new op lower bound from 7316292849018966016 to 7316292848851718144. Current Snapshot: MvccSnapshot[applied={T|T < 7316292849018966016}]
I20260808 15:51:48.891250 17074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.891515 17074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.897626 16923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593837f757474ef4993034d411803eb4, State: Initialized, Role: FOLLOWER
I20260808 15:51:48.898491 16923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } }
I20260808 15:51:48.898748 17103 mvcc.cc:205] Tried to move back new op lower bound from 7316292849018966016 to 7316292848851718144. Current Snapshot: MvccSnapshot[applied={T|T < 7316292849018966016 or (T in {7316292849018966016})}]
I20260808 15:51:48.903260 17044 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.903578 17044 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:48.904619 17099 catalog_manager.cc:1347] Generated new cluster ID: 9bf2ba7288644848b73762ae3c8a2878
I20260808 15:51:48.904733 17099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:51:48.909803 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593837f757474ef4993034d411803eb4" is_pre_election: true
I20260808 15:51:48.910435 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593837f757474ef4993034d411803eb4"
I20260808 15:51:48.911188 16880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:48.916033 16923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:48.921041 16880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [term 1 FOLLOWER]: Refusing update from remote peer b5cdfc3baaf04bb6a6209c0cb8004f30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:51:48.922236 17094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:51:48.902475 17108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.925257 17108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.936403 17099 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:51:48.939023 17099 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:51:48.974236 17111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.974785 17111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.974963 17108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5cdfc3baaf04bb6a6209c0cb8004f30. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.975219 17108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:51:48.976791 17094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5cdfc3baaf04bb6a6209c0cb8004f30" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36031 } } peers { permanent_uuid: "593837f757474ef4993034d411803eb4" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41443 } } peers { permanent_uuid: "6465fc4f712341059594d57095bbdcb7" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 40991 } } }
I20260808 15:51:48.977201 17094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:48.981639 17099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5cdfc3baaf04bb6a6209c0cb8004f30: Generated new TSK 0
I20260808 15:51:48.982972 17099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:51:49.010004 17132 catalog_manager.cc:1259] Loaded cluster ID: 9bf2ba7288644848b73762ae3c8a2878
I20260808 15:51:49.010248 17132 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: loading cluster ID for follower catalog manager: success
I20260808 15:51:49.014458 17132 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: acquiring CA information for follower catalog manager: success
I20260808 15:51:49.017284 17132 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 593837f757474ef4993034d411803eb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:51:49.799563 17092 catalog_manager.cc:1259] Loaded cluster ID: 9bf2ba7288644848b73762ae3c8a2878
I20260808 15:51:49.799706 17092 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: loading cluster ID for follower catalog manager: success
I20260808 15:51:49.802269 17092 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: acquiring CA information for follower catalog manager: success
I20260808 15:51:49.804193 17092 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6465fc4f712341059594d57095bbdcb7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:51:50.139235 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 16696
I20260808 15:51:50.140530 16801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:50.254869 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 16696
I20260808 15:51:50.287217 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 16817
I20260808 15:51:50.288380 16915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:50.396687 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 16817
I20260808 15:51:50.426190 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 16932
I20260808 15:51:50.427569 17022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:50.538316 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 16932
2026-08-08T15:51:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5460 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:51:50 dist-test-slave-twwt krb5kdc[17148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:51:50 dist-test-slave-twwt krb5kdc[17148](info): set up 2 sockets
Aug 08 15:51:50 dist-test-slave-twwt krb5kdc[17148](info): commencing operation
krb5kdc: starting...
W20260808 15:51:51.876799 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.004s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:51:52 dist-test-slave-twwt krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:51:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:51:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:51:52.194492 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:38653
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:52.552562 17167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:52.552940 17167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:52.553114 17167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:52.560472 17167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:52.560580 17167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:52.560688 17167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:52.560803 17167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:52.560879 17167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:52.571010 17167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:52.572429 17167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:52.573818 17167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:52.586400 17177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:52.586419 17176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:52.586784 17179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:52.587384 17167 server_base.cc:1034] running on GCE node
I20260808 15:51:52.588234 17167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:52.589812 17167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:52.591053 17167 hybrid_clock.cc:648] HybridClock initialized: now 1786204312590996 us; error 45 us; skew 500 ppm
I20260808 15:51:52.591439 17167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:52 dist-test-slave-twwt krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:52.594738 17167 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:51:52.596344 17167 webserver.cc:460] Webserver started at http://127.27.84.254:42455/ using document root <none> and password file <none>
I20260808 15:51:52.597049 17167 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:52.597196 17167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:52.597512 17167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:52.600630 17167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "60af7632c09442418d0ac74a05c809a8"
format_stamp: "Formatted at 2026-08-08 15:51:52 on dist-test-slave-twwt"
server_key: "9be6212ddf1d2f2166f2e4ec2d720ea2"
server_key_iv: "7f2b9049e6904bcd82107fbc438a9f0b"
server_key_version: "encryptionkey@0"
I20260808 15:51:52.601410 17167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "60af7632c09442418d0ac74a05c809a8"
format_stamp: "Formatted at 2026-08-08 15:51:52 on dist-test-slave-twwt"
server_key: "9be6212ddf1d2f2166f2e4ec2d720ea2"
server_key_iv: "7f2b9049e6904bcd82107fbc438a9f0b"
server_key_version: "encryptionkey@0"
I20260808 15:51:52.606724 17167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:51:52.610131 17190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:52.611523 17167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:51:52.611708 17167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "60af7632c09442418d0ac74a05c809a8"
format_stamp: "Formatted at 2026-08-08 15:51:52 on dist-test-slave-twwt"
server_key: "9be6212ddf1d2f2166f2e4ec2d720ea2"
server_key_iv: "7f2b9049e6904bcd82107fbc438a9f0b"
server_key_version: "encryptionkey@0"
I20260808 15:51:52.611907 17167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:52.628499 17167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:52.632576 17167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:52.632885 17167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:52.652271 17167 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:38653
I20260808 15:51:52.652313 17265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:38653 every 8 connection(s)
I20260808 15:51:52.653774 17167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:51:52.656517 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 17167
I20260808 15:51:52.656785 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:51:52.657706 27987 external_mini_cluster.cc:1428] Setting key b1cc0b07f537050b4cd8cec607582488
I20260808 15:51:52.658764 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:51:52 dist-test-slave-twwt krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:52.674008 17266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8: Bootstrap starting.
I20260808 15:51:52.677628 17266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:52.678840 17266 log.cc:824] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:52.681915 17266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8: No bootstrap required, opened a new log
I20260808 15:51:52.682919 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:52.659738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.27.84.254:38653)
0808 15:51:52.660548 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:52.660560 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:52.660601 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:51:52.662245 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:52.662268 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:52.662759 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:52.663132 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:52.665193 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:52.666909 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:52.668825 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:52.669665 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:52.669850 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:52.675235 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:52.675243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:52.675259 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:52.675299 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:52.678544 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:52.679407 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:52.679419 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:52.679425 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:52.679638 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:52.680028 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:52.680037 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:52.680043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:52.680575 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:52.680897 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:52.681376 (+ 479us) server_negotiation.cc:299] Negotiation successful
0808 15:51:52.681827 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":366,"threads_started":1}
I20260808 15:51:52.686896 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:51:52.690541 17266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60af7632c09442418d0ac74a05c809a8" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38653 } }
I20260808 15:51:52.690992 17266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:52.691105 17266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60af7632c09442418d0ac74a05c809a8, State: Initialized, Role: FOLLOWER
I20260808 15:51:52.691682 17266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60af7632c09442418d0ac74a05c809a8" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38653 } }
I20260808 15:51:52.691936 17266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:51:52.692087 17266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:51:52.692291 17266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:52.694654 17266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60af7632c09442418d0ac74a05c809a8" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38653 } }
I20260808 15:51:52.695217 17266 leader_election.cc:302] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60af7632c09442418d0ac74a05c809a8; no voters:
I20260808 15:51:52.695866 17266 leader_election.cc:288] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:51:52.696039 17273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:51:52.696789 17273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [term 1 LEADER]: Becoming Leader. State: Replica: 60af7632c09442418d0ac74a05c809a8, State: Running, Role: LEADER
I20260808 15:51:52.697324 17273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60af7632c09442418d0ac74a05c809a8" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38653 } }
I20260808 15:51:52.698053 17266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:52.700659 17275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60af7632c09442418d0ac74a05c809a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60af7632c09442418d0ac74a05c809a8" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38653 } } }
I20260808 15:51:52.700956 17275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:52.701304 17276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60af7632c09442418d0ac74a05c809a8. Latest consensus state: current_term: 1 leader_uuid: "60af7632c09442418d0ac74a05c809a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60af7632c09442418d0ac74a05c809a8" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 38653 } } }
I20260808 15:51:52.701716 17276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:52.702014 17284 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:51:52.709100 17284 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:51:52.723201 17284 catalog_manager.cc:1347] Generated new cluster ID: f868afe355274e11b2a666c11ba38125
I20260808 15:51:52.723358 17284 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:51:52.758562 17284 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:51:52.760568 17284 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:51:52.777295 17284 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60af7632c09442418d0ac74a05c809a8: Generated new TSK 0
I20260808 15:51:52.778426 17284 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:51:52.799626 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 17167
I20260808 15:51:52.800806 17256 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:52.898308 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 17167
2026-08-08T15:51:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2363 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:51:53 dist-test-slave-twwt krb5kdc[17314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:51:53 dist-test-slave-twwt krb5kdc[17314](info): set up 2 sockets
Aug 08 15:51:53 dist-test-slave-twwt krb5kdc[17314](info): commencing operation
krb5kdc: starting...
W20260808 15:51:54.251945 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.302s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:51:54 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:51:54.582808 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43565
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:54.931524 17339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:54.931856 17339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:54.932014 17339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:54.939491 17339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:54.939651 17339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:54.939754 17339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:54.939853 17339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:54.939934 17339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:54.950441 17339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:54.951905 17339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:54.953305 17339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:54.965324 17346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:54.965413 17349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:54.965355 17347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:54.966444 17339 server_base.cc:1034] running on GCE node
I20260808 15:51:54.967231 17339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:54.968643 17339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:54.969851 17339 hybrid_clock.cc:648] HybridClock initialized: now 1786204314969801 us; error 39 us; skew 500 ppm
I20260808 15:51:54.970191 17339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:54 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:54.973492 17339 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:51:54.975116 17339 webserver.cc:460] Webserver started at http://127.27.84.254:45123/ using document root <none> and password file <none>
I20260808 15:51:54.975778 17339 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:54.975885 17339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:54.976145 17339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:54.979684 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "74595f1069fd46f4abb46201ac4d3699"
format_stamp: "Formatted at 2026-08-08 15:51:54 on dist-test-slave-twwt"
server_key: "b164f7dc8c88a737bff707f6804088ce"
server_key_iv: "c9653c81cdcb7b1460302bea4e392368"
server_key_version: "encryptionkey@0"
I20260808 15:51:54.980461 17339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "74595f1069fd46f4abb46201ac4d3699"
format_stamp: "Formatted at 2026-08-08 15:51:54 on dist-test-slave-twwt"
server_key: "b164f7dc8c88a737bff707f6804088ce"
server_key_iv: "c9653c81cdcb7b1460302bea4e392368"
server_key_version: "encryptionkey@0"
I20260808 15:51:54.985932 17339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:51:54.989713 17361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:54.991434 17339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:51:54.991622 17339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "74595f1069fd46f4abb46201ac4d3699"
format_stamp: "Formatted at 2026-08-08 15:51:54 on dist-test-slave-twwt"
server_key: "b164f7dc8c88a737bff707f6804088ce"
server_key_iv: "c9653c81cdcb7b1460302bea4e392368"
server_key_version: "encryptionkey@0"
I20260808 15:51:54.991817 17339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:55.021592 17339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:55.026295 17339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:55.026628 17339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:55.046684 17339 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43565
I20260808 15:51:55.046685 17444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43565 every 8 connection(s)
I20260808 15:51:55.048202 17339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:51:55.052246 17445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:55.053148 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 17339
I20260808 15:51:55.053426 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:51:55.054347 27987 external_mini_cluster.cc:1428] Setting key 9b4eddf6a6a28d1d95dd2ddcaa6aa2e4
I20260808 15:51:55.067826 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap starting.
Aug 08 15:51:55 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:55.071806 17445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:55.073321 17445 log.cc:824] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:55.076942 17445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: No bootstrap required, opened a new log
I20260808 15:51:55.080094 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:55.056696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.27.84.254:43565)
0808 15:51:55.057533 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:55.057545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:55.057590 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:51:55.059346 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:55.059370 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:55.059801 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:55.060170 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:55.061993 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:55.063483 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:55.065352 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:55.066025 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:55.066168 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:55.071197 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:55.071208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:55.071227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:55.071279 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:55.075284 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:55.076056 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:55.076067 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:55.076078 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:55.076295 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:55.076685 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:55.076690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:55.076693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:55.077426 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:55.077771 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:55.078139 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 15:51:55.078647 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":603,"thread_start_us":369,"threads_started":1}
I20260808 15:51:55.086189 17445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:55.086566 17445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:55.086701 17445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74595f1069fd46f4abb46201ac4d3699, State: Initialized, Role: FOLLOWER
I20260808 15:51:55.087314 17445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:55.087563 17445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:51:55.087719 17445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:51:55.087900 17445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:55.090291 17445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:55.090802 17445 leader_election.cc:302] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74595f1069fd46f4abb46201ac4d3699; no voters:
I20260808 15:51:55.091403 17445 leader_election.cc:288] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:51:55.091548 17455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:51:55.092278 17455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 LEADER]: Becoming Leader. State: Replica: 74595f1069fd46f4abb46201ac4d3699, State: Running, Role: LEADER
I20260808 15:51:55.092837 17455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:55.093711 17445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:55.096201 17457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74595f1069fd46f4abb46201ac4d3699. Latest consensus state: current_term: 1 leader_uuid: "74595f1069fd46f4abb46201ac4d3699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } } }
I20260808 15:51:55.096145 17456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74595f1069fd46f4abb46201ac4d3699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } } }
I20260808 15:51:55.096702 17456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:55.097548 17457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:55.099027 17461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:51:55.103597 17461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:51:55.112569 17461 catalog_manager.cc:1347] Generated new cluster ID: 3f7df20644db4b49a2010e8ecc9b7fce
I20260808 15:51:55.112721 17461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:51:55.127209 17461 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:51:55.128491 17461 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:51:55.135846 17461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Generated new TSK 0
I20260808 15:51:55.136965 17461 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:51:55.217577 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43565
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:55.591125 17487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:55.591418 17487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:55.591609 17487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:55.599753 17487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:55.599859 17487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:55.600036 17487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:51:55.609926 17487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:55.611351 17487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:55.612777 17487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:55.626394 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:55.626557 17502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:55.627177 17504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:55.627571 17487 server_base.cc:1034] running on GCE node
I20260808 15:51:55.628196 17487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:55.629196 17487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:55.630417 17487 hybrid_clock.cc:648] HybridClock initialized: now 1786204315630358 us; error 56 us; skew 500 ppm
I20260808 15:51:55.630786 17487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:55 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:55.633991 17487 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:51:55.635689 17487 webserver.cc:460] Webserver started at http://127.27.84.193:36023/ using document root <none> and password file <none>
I20260808 15:51:55.636421 17487 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:55.636571 17487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:55.636890 17487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:55.640152 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "aff13f6354df4055a769f2418f28398b"
format_stamp: "Formatted at 2026-08-08 15:51:55 on dist-test-slave-twwt"
server_key: "18b228c1ebdbb321101ecc65c576ed50"
server_key_iv: "d9ceb652ab13fa9975afe3b6ffcfdabe"
server_key_version: "encryptionkey@0"
I20260808 15:51:55.641010 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "aff13f6354df4055a769f2418f28398b"
format_stamp: "Formatted at 2026-08-08 15:51:55 on dist-test-slave-twwt"
server_key: "18b228c1ebdbb321101ecc65c576ed50"
server_key_iv: "d9ceb652ab13fa9975afe3b6ffcfdabe"
server_key_version: "encryptionkey@0"
I20260808 15:51:55.646401 17487 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:51:55.650107 17513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:55.651616 17487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:51:55.651798 17487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "aff13f6354df4055a769f2418f28398b"
format_stamp: "Formatted at 2026-08-08 15:51:55 on dist-test-slave-twwt"
server_key: "18b228c1ebdbb321101ecc65c576ed50"
server_key_iv: "d9ceb652ab13fa9975afe3b6ffcfdabe"
server_key_version: "encryptionkey@0"
I20260808 15:51:55.652016 17487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:55.673878 17487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:55.679050 17487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:55.679383 17487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:55.680455 17487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:55.682013 17487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:55.682120 17487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:55.682279 17487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:55.682361 17487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:55.717782 17487 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43453
I20260808 15:51:55.717881 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43453 every 8 connection(s)
I20260808 15:51:55.719298 17487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:51:55.729420 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 17487
I20260808 15:51:55.729655 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:51:55.730536 27987 external_mini_cluster.cc:1428] Setting key 329802ebc1f1990b3a34e64fef5cc77a
Aug 08 15:51:55 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:55.742877 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:55.722360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55039 (local address 127.27.84.254:43565)
0808 15:51:55.722897 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:55.722903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:55.723991 (+ 1088us) server_negotiation.cc:407] Connection header received
0808 15:51:55.725899 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:55.725904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:55.725982 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:55.726097 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:55.727813 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:55.728588 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:55.731117 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:55.731619 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:55.731767 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:55.736274 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:55.736280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:55.736283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:55.736333 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:55.739086 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:55.740170 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:55.740176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:55.740178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:55.740329 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:55.740913 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:55.740917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:55.740919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:55.741213 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:55.741333 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:55.742294 (+ 961us) server_negotiation.cc:299] Negotiation successful
0808 15:51:55.742487 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260808 15:51:55.744448 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:55.723031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:55039)
0808 15:51:55.723806 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:55.723864 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:55.725597 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:55.726318 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:55.726326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:55.726999 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:55.727621 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:55.727640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:55.728745 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:55.728748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:55.730956 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:55.730974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:55.731950 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:55.731959 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:55.732256 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:55.733085 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:55.733110 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:55.736005 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:55.739236 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:55.739249 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:55.739272 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:55.739994 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:55.740463 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:55.740471 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:55.740478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:55.740780 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:55.741352 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:55.741363 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:55.741868 (+ 505us) client_negotiation.cc:769] Sending connection context
0808 15:51:55.742193 (+ 325us) client_negotiation.cc:240] Negotiation successful
0808 15:51:55.742857 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":303,"threads_started":1}
I20260808 15:51:55.746373 17679 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43565
I20260808 15:51:55.746714 17679 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:55.747463 17679 heartbeater.cc:511] Master 127.27.84.254:43565 requested a full tablet report, sending...
I20260808 15:51:55.749694 17391 ts_manager.cc:194] Registered new tserver with Master: aff13f6354df4055a769f2418f28398b (127.27.84.193:43453)
I20260808 15:51:55.751269 17391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:55039
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:51:55.802048 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43565
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:56.163862 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:56.164155 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:56.164253 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:56.171664 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:56.171797 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:56.171912 17690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:51:56.182168 17690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:56.183583 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:56.184989 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:56.199201 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:56.199172 17701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:56.199132 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:56.200234 17690 server_base.cc:1034] running on GCE node
I20260808 15:51:56.200913 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:56.201858 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:56.203063 17690 hybrid_clock.cc:648] HybridClock initialized: now 1786204316202987 us; error 61 us; skew 500 ppm
I20260808 15:51:56.203409 17690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:56 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:56.207019 17690 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:51:56.208638 17690 webserver.cc:460] Webserver started at http://127.27.84.194:33461/ using document root <none> and password file <none>
I20260808 15:51:56.209431 17690 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:56.209570 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:56.209919 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:56.213191 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "a62d539f6b9846359bbd85ecdcbbf327"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "e0e4de4c05b385a8cf6816216a8783ca"
server_key_iv: "9d3e6a621b32712475a0095273a9829e"
server_key_version: "encryptionkey@0"
I20260808 15:51:56.214059 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "a62d539f6b9846359bbd85ecdcbbf327"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "e0e4de4c05b385a8cf6816216a8783ca"
server_key_iv: "9d3e6a621b32712475a0095273a9829e"
server_key_version: "encryptionkey@0"
I20260808 15:51:56.219825 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:51:56.223634 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:56.225193 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:51:56.225391 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "a62d539f6b9846359bbd85ecdcbbf327"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "e0e4de4c05b385a8cf6816216a8783ca"
server_key_iv: "9d3e6a621b32712475a0095273a9829e"
server_key_version: "encryptionkey@0"
I20260808 15:51:56.225620 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:56.247514 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:56.252310 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:56.252638 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:56.253862 17690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:56.255522 17690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:56.255601 17690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:56.255795 17690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:56.255877 17690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:56.292822 17690 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:42115
I20260808 15:51:56.292867 17880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:42115 every 8 connection(s)
I20260808 15:51:56.294468 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:51:56.304397 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 17690
I20260808 15:51:56.304594 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:51:56.305068 27987 external_mini_cluster.cc:1428] Setting key cacef4662f99af82e5423c0b40ada9e0
Aug 08 15:51:56 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:56.315804 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.297447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:44371 (local address 127.27.84.254:43565)
0808 15:51:56.297960 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:56.297965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:56.299191 (+ 1226us) server_negotiation.cc:407] Connection header received
0808 15:51:56.301133 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:56.301139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:56.301235 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:56.301360 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:56.303048 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.303799 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.305644 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.306114 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.306265 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.309850 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:56.309856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:56.309860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:56.309892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:56.312646 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:56.313396 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:56.313406 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:56.313408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:56.313543 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:56.313998 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:56.314002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:56.314004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:56.314272 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:56.314432 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:56.315187 (+ 755us) server_negotiation.cc:299] Negotiation successful
0808 15:51:56.315418 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":240,"threads_started":1}
I20260808 15:51:56.316830 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.298135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:44371)
0808 15:51:56.299011 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:56.299056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:56.300825 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:56.301634 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:56.301644 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:56.302192 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:56.302824 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.302844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.303961 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.303967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.305497 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.305510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.306368 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.306379 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.306672 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.307376 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:56.307405 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:56.309616 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:56.312793 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:56.312802 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:56.312819 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:56.313251 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:56.313658 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:56.313663 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:56.313667 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:56.313869 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:56.314404 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:56.314412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:56.314814 (+ 402us) client_negotiation.cc:769] Sending connection context
0808 15:51:56.315097 (+ 283us) client_negotiation.cc:240] Negotiation successful
0808 15:51:56.315670 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":299,"threads_started":1}
I20260808 15:51:56.318464 17882 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43565
I20260808 15:51:56.318799 17882 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:56.319589 17882 heartbeater.cc:511] Master 127.27.84.254:43565 requested a full tablet report, sending...
I20260808 15:51:56.321107 17391 ts_manager.cc:194] Registered new tserver with Master: a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194:42115)
I20260808 15:51:56.322014 17391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:44371
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:51:56.375113 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43565
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:56.737140 17890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:56.737433 17890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:56.737529 17890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:56.744848 17890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:56.744973 17890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:56.745088 17890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:51:56.754345 17679 heartbeater.cc:503] Master 127.27.84.254:43565 was elected leader, sending a full tablet report...
I20260808 15:51:56.755357 17890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:56.756973 17890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:56.758332 17890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:56.771859 17899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:56.771950 17901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:56.771859 17897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:56.772732 17890 server_base.cc:1034] running on GCE node
I20260808 15:51:56.773307 17890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:56.774219 17890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:56.775471 17890 hybrid_clock.cc:648] HybridClock initialized: now 1786204316775410 us; error 52 us; skew 500 ppm
I20260808 15:51:56.775856 17890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:56 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:56.779213 17890 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:51:56.780702 17890 webserver.cc:460] Webserver started at http://127.27.84.195:45041/ using document root <none> and password file <none>
I20260808 15:51:56.781407 17890 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:56.781556 17890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:56.781886 17890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:51:56.784942 17890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "e2d533efa1c44386a1d472cf08e91b13"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "7c22f5cb880fd45daad4c02f970b6015"
server_key_iv: "2fdc0ea629cca08fcb270534fb2bf0f1"
server_key_version: "encryptionkey@0"
I20260808 15:51:56.785786 17890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "e2d533efa1c44386a1d472cf08e91b13"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "7c22f5cb880fd45daad4c02f970b6015"
server_key_iv: "2fdc0ea629cca08fcb270534fb2bf0f1"
server_key_version: "encryptionkey@0"
I20260808 15:51:56.791231 17890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260808 15:51:56.795095 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:56.796613 17890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:51:56.796780 17890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "e2d533efa1c44386a1d472cf08e91b13"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "7c22f5cb880fd45daad4c02f970b6015"
server_key_iv: "2fdc0ea629cca08fcb270534fb2bf0f1"
server_key_version: "encryptionkey@0"
I20260808 15:51:56.797008 17890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:56.818094 17890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:56.822458 17890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:56.822805 17890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:56.823794 17890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:56.825173 17890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:56.825287 17890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:56.825429 17890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:56.825498 17890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:56.861114 17890 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:44933
I20260808 15:51:56.861166 18081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:44933 every 8 connection(s)
I20260808 15:51:56.862495 17890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:51:56.867882 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 17890
I20260808 15:51:56.868085 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:51:56.868548 27987 external_mini_cluster.cc:1428] Setting key 5608dfe1a225fe7780feea05bd214a3f
Aug 08 15:51:56 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204316, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:56.884313 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.865234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:33155 (local address 127.27.84.254:43565)
0808 15:51:56.865726 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:56.865731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:56.866820 (+ 1089us) server_negotiation.cc:407] Connection header received
0808 15:51:56.868831 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:56.868836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:56.868941 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:56.869066 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:56.870896 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.871926 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.873629 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.874019 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.874121 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.877977 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:56.877982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:56.877986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:56.878023 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:56.880658 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:56.881493 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:56.881497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:56.881500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:56.881623 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:56.882082 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:56.882087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:56.882090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:56.882482 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:56.882670 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:56.883724 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0808 15:51:56.883984 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":232,"threads_started":1}
I20260808 15:51:56.885607 18094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.865831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:33155)
0808 15:51:56.866575 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:56.866676 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:56.868524 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:56.869328 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:56.869337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:56.870022 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:56.870701 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.870733 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.872079 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.872083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.873482 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.873496 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.874273 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.874279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.874504 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.875289 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:56.875316 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:56.877742 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:56.880823 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:56.880832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:56.880847 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:56.881364 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:56.881721 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:56.881725 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:56.881727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:56.881946 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:56.882665 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:56.882677 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:56.883227 (+ 550us) client_negotiation.cc:769] Sending connection context
0808 15:51:56.883565 (+ 338us) client_negotiation.cc:240] Negotiation successful
0808 15:51:56.884165 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":283,"threads_started":1}
I20260808 15:51:56.887003 18084 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43565
I20260808 15:51:56.887327 18084 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:56.888160 18084 heartbeater.cc:511] Master 127.27.84.254:43565 requested a full tablet report, sending...
I20260808 15:51:56.889792 17391 ts_manager.cc:194] Registered new tserver with Master: e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933)
I20260808 15:51:56.890787 17391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:33155
I20260808 15:51:56.897367 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:51:56.915675 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.902477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.27.84.254:43565)
0808 15:51:56.902720 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:56.902726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:56.903169 (+ 443us) server_negotiation.cc:407] Connection header received
0808 15:51:56.903245 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:56.903249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:56.903347 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:56.903489 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:56.904897 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.905708 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.907086 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.907524 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.907620 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.910107 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:56.910113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:56.910116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:56.910152 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:56.912950 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:56.913687 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:56.913691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:56.913693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:56.913815 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:56.914152 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:56.914155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:56.914158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:56.914492 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:56.914756 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:56.915211 (+ 455us) server_negotiation.cc:299] Negotiation successful
0808 15:51:56.915430 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:51:56.920499 17391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:51:56.928349 17391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:51:56.978999 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.942946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:44933 (local address 127.0.0.1:60726)
0808 15:51:56.968070 (+ 25124us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:56.968095 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:56.968253 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:56.969763 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:56.969777 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:56.969834 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:56.970450 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.970470 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.973249 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.973255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.975711 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.975731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.977839 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.977846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.978057 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.978118 (+ 61us) client_negotiation.cc:769] Sending connection context
0808 15:51:56.978299 (+ 181us) client_negotiation.cc:240] Negotiation successful
0808 15:51:56.978524 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24954,"mutex_wait_us":107,"thread_start_us":26642,"threads_started":1}
I20260808 15:51:56.979131 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.943542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.27.84.195:44933)
0808 15:51:56.966309 (+ 22767us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:56.966320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:56.968425 (+ 2105us) server_negotiation.cc:407] Connection header received
0808 15:51:56.968594 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:56.968605 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:56.968976 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:56.969261 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:56.970774 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.973091 (+ 2317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.975928 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.977627 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.977802 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.978065 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:56.978525 (+ 460us) server_negotiation.cc:299] Negotiation successful
0808 15:51:56.978776 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22502,"thread_start_us":22795,"threads_started":1}
I20260808 15:51:56.981273 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.942447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42115 (local address 127.0.0.1:50932)
0808 15:51:56.951849 (+ 9402us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:56.951912 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:56.952187 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:56.971074 (+ 18887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:56.971079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:56.971115 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:56.971781 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.971805 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.973882 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.973888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.977681 (+ 3793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.977696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.980201 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.980212 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.980419 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.980462 (+ 43us) client_negotiation.cc:769] Sending connection context
0808 15:51:56.980546 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:51:56.980763 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9008,"mutex_wait_us":422,"thread_start_us":8803,"threads_started":1}
I20260808 15:51:56.981455 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.942449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43453 (local address 127.0.0.1:38914)
0808 15:51:56.967171 (+ 24722us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:56.967202 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:56.967434 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:56.971462 (+ 4028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:56.971467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:56.971493 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:56.972100 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.972114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.974047 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.974055 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.977677 (+ 3622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:56.977696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.980511 (+ 2815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.980516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:56.980740 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.980792 (+ 52us) client_negotiation.cc:769] Sending connection context
0808 15:51:56.980888 (+ 96us) client_negotiation.cc:240] Negotiation successful
0808 15:51:56.981030 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24525,"mutex_wait_us":509,"thread_start_us":25036,"threads_started":1}
I20260808 15:51:56.981802 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.943022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.27.84.194:42115)
0808 15:51:56.970376 (+ 27354us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:56.970387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:56.970420 (+ 33us) server_negotiation.cc:407] Connection header received
0808 15:51:56.970537 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:56.970547 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:56.970831 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:56.971068 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:56.972138 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.973721 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.977943 (+ 4222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.979933 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.980742 (+ 809us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.981009 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:56.981084 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 15:51:56.981275 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27164,"thread_start_us":27028,"threads_started":1}
I20260808 15:51:56.982993 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:56.942816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.27.84.193:43453)
0808 15:51:56.970841 (+ 28025us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:56.970849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:56.970878 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:51:56.970989 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:56.971000 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:56.971226 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:56.971444 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:56.972346 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.973876 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.977967 (+ 4091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:56.979697 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:56.979888 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:56.980186 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:56.981700 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0808 15:51:56.981911 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27830,"thread_start_us":27650,"threads_started":1}
I20260808 15:51:56.984472 17987 tablet_service.cc:1467] Processing CreateTablet for tablet e0a03ad3c2b641e4b27ad662e5963eaf (DEFAULT_TABLE table=test-table [id=daa518193c75423290afb01d9f7df085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:56.986271 17987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a03ad3c2b641e4b27ad662e5963eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:56.985941 17782 tablet_service.cc:1467] Processing CreateTablet for tablet e0a03ad3c2b641e4b27ad662e5963eaf (DEFAULT_TABLE table=test-table [id=daa518193c75423290afb01d9f7df085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:56.987824 17782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a03ad3c2b641e4b27ad662e5963eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:56.988543 17583 tablet_service.cc:1467] Processing CreateTablet for tablet e0a03ad3c2b641e4b27ad662e5963eaf (DEFAULT_TABLE table=test-table [id=daa518193c75423290afb01d9f7df085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:56.989681 17583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a03ad3c2b641e4b27ad662e5963eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:57.001471 18116 tablet_bootstrap.cc:492] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Bootstrap starting.
I20260808 15:51:57.005106 18116 tablet_bootstrap.cc:654] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:57.005676 18119 tablet_bootstrap.cc:492] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Bootstrap starting.
I20260808 15:51:57.006783 18116 log.cc:824] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:57.009840 18119 tablet_bootstrap.cc:654] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:57.008512 18118 tablet_bootstrap.cc:492] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Bootstrap starting.
I20260808 15:51:57.010923 18116 tablet_bootstrap.cc:492] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: No bootstrap required, opened a new log
I20260808 15:51:57.011252 18116 ts_tablet_manager.cc:1397] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 15:51:57.011413 18119 log.cc:824] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:57.014675 18118 tablet_bootstrap.cc:654] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:57.015092 18119 tablet_bootstrap.cc:492] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: No bootstrap required, opened a new log
I20260808 15:51:57.015417 18119 ts_tablet_manager.cc:1397] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260808 15:51:57.015975 18118 log.cc:824] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:57.018999 18118 tablet_bootstrap.cc:492] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: No bootstrap required, opened a new log
I20260808 15:51:57.019332 18118 ts_tablet_manager.cc:1397] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260808 15:51:57.022266 18116 raft_consensus.cc:348] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.022815 18116 raft_consensus.cc:374] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:57.022939 18116 raft_consensus.cc:729] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff13f6354df4055a769f2418f28398b, State: Initialized, Role: FOLLOWER
I20260808 15:51:57.023675 18116 consensus_queue.cc:260] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.025661 18116 ts_tablet_manager.cc:1428] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260808 15:51:57.028501 18119 raft_consensus.cc:348] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.029071 18119 raft_consensus.cc:374] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:57.029193 18119 raft_consensus.cc:729] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d533efa1c44386a1d472cf08e91b13, State: Initialized, Role: FOLLOWER
I20260808 15:51:57.030037 18119 consensus_queue.cc:260] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.030158 18118 raft_consensus.cc:348] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.030876 18118 raft_consensus.cc:374] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:57.031001 18118 raft_consensus.cc:729] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62d539f6b9846359bbd85ecdcbbf327, State: Initialized, Role: FOLLOWER
I20260808 15:51:57.031571 18084 heartbeater.cc:503] Master 127.27.84.254:43565 was elected leader, sending a full tablet report...
I20260808 15:51:57.031713 18118 consensus_queue.cc:260] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.032316 18119 ts_tablet_manager.cc:1428] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260808 15:51:57.033356 17882 heartbeater.cc:503] Master 127.27.84.254:43565 was elected leader, sending a full tablet report...
I20260808 15:51:57.034052 18118 ts_tablet_manager.cc:1428] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
I20260808 15:51:57.039181 18125 raft_consensus.cc:482] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:51:57.039453 18125 raft_consensus.cc:504] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.041265 18125 leader_election.cc:288] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aff13f6354df4055a769f2418f28398b (127.27.84.193:43453), e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933)
W20260808 15:51:57.046741 17884 tablet.cc:2369] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:51:57.049254 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.041689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:58383 (local address 127.27.84.193:43453)
0808 15:51:57.042013 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:57.042024 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:57.043091 (+ 1067us) server_negotiation.cc:407] Connection header received
0808 15:51:57.043326 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:57.043332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:57.043400 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:57.043522 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:57.044124 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.044893 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.047249 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.048213 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.048316 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.048379 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:57.048872 (+ 493us) server_negotiation.cc:299] Negotiation successful
0808 15:51:57.049001 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 15:51:57.049432 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.041538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43453 (local address 127.27.84.194:58383)
0808 15:51:57.042942 (+ 1404us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:57.042974 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:57.043151 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:57.043555 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:57.043560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:57.043588 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:57.043968 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:57.043980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.045026 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.045032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:57.047046 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:57.047062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.048584 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.048595 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:57.048707 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.048731 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:51:57.048839 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 15:51:57.048976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":48,"thread_start_us":1328,"threads_started":1}
I20260808 15:51:57.050031 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.042573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:49609 (local address 127.27.84.195:44933)
0808 15:51:57.042785 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:57.042790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:57.043326 (+ 536us) server_negotiation.cc:407] Connection header received
0808 15:51:57.043500 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:57.043504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:57.043565 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:57.043658 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:57.044321 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.045050 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.047432 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.048647 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.048775 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.048852 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:57.049622 (+ 770us) server_negotiation.cc:299] Negotiation successful
0808 15:51:57.049772 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:51:57.050535 17612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a03ad3c2b641e4b27ad662e5963eaf" candidate_uuid: "a62d539f6b9846359bbd85ecdcbbf327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff13f6354df4055a769f2418f28398b" is_pre_election: true
I20260808 15:51:57.050781 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.041771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:44933 (local address 127.27.84.194:49609)
0808 15:51:57.043232 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:57.043250 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:57.043363 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:57.043678 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:57.043682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:57.043727 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:57.044165 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:57.044175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.045202 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.045215 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:57.047248 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:57.047265 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.049045 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.049051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:57.049242 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.049275 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:51:57.050254 (+ 979us) client_negotiation.cc:240] Negotiation successful
0808 15:51:57.050395 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1329,"mutex_wait_us":13,"thread_start_us":440,"threads_started":1}
I20260808 15:51:57.051196 17612 raft_consensus.cc:2454] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62d539f6b9846359bbd85ecdcbbf327 in term 0.
I20260808 15:51:57.051755 18017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a03ad3c2b641e4b27ad662e5963eaf" candidate_uuid: "a62d539f6b9846359bbd85ecdcbbf327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d533efa1c44386a1d472cf08e91b13" is_pre_election: true
I20260808 15:51:57.051976 17710 leader_election.cc:302] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62d539f6b9846359bbd85ecdcbbf327, aff13f6354df4055a769f2418f28398b; no voters:
I20260808 15:51:57.052306 18017 raft_consensus.cc:2454] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62d539f6b9846359bbd85ecdcbbf327 in term 0.
I20260808 15:51:57.052577 18125 raft_consensus.cc:2781] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:51:57.052845 18125 raft_consensus.cc:482] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:51:57.052975 18125 raft_consensus.cc:3037] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:57.055307 18125 raft_consensus.cc:504] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.056016 18125 leader_election.cc:288] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [CANDIDATE]: Term 1 election: Requested vote from peers aff13f6354df4055a769f2418f28398b (127.27.84.193:43453), e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933)
I20260808 15:51:57.056519 18017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a03ad3c2b641e4b27ad662e5963eaf" candidate_uuid: "a62d539f6b9846359bbd85ecdcbbf327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d533efa1c44386a1d472cf08e91b13"
I20260808 15:51:57.056761 18017 raft_consensus.cc:3037] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:57.056725 17612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a03ad3c2b641e4b27ad662e5963eaf" candidate_uuid: "a62d539f6b9846359bbd85ecdcbbf327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aff13f6354df4055a769f2418f28398b"
I20260808 15:51:57.056941 17612 raft_consensus.cc:3037] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:51:57.059201 18017 raft_consensus.cc:2454] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62d539f6b9846359bbd85ecdcbbf327 in term 1.
I20260808 15:51:57.059636 17712 leader_election.cc:302] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a62d539f6b9846359bbd85ecdcbbf327, e2d533efa1c44386a1d472cf08e91b13; no voters:
I20260808 15:51:57.059923 18125 raft_consensus.cc:2781] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:51:57.060130 17612 raft_consensus.cc:2454] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62d539f6b9846359bbd85ecdcbbf327 in term 1.
I20260808 15:51:57.060544 18125 raft_consensus.cc:686] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 1 LEADER]: Becoming Leader. State: Replica: a62d539f6b9846359bbd85ecdcbbf327, State: Running, Role: LEADER
I20260808 15:51:57.061064 18125 consensus_queue.cc:237] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } }
I20260808 15:51:57.065630 17390 catalog_manager.cc:5495] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 reported cstate change: term changed from 0 to 1, leader changed from <none> to a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "a62d539f6b9846359bbd85ecdcbbf327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } health_report { overall_health: UNKNOWN } } }
I20260808 15:51:57.083892 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.078392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50942 (local address 127.27.84.194:42115)
0808 15:51:57.078599 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:57.078631 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:57.078740 (+ 109us) server_negotiation.cc:407] Connection header received
0808 15:51:57.078984 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:57.078988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:57.079073 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:57.079219 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:51:57.079608 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.080329 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.081724 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.082138 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.082271 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.082773 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:51:57.083284 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:51:57.083450 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:57.083534 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 15:51:57.083615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:51:57.092258 18017 raft_consensus.cc:1261] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Refusing update from remote peer a62d539f6b9846359bbd85ecdcbbf327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:51:57.092432 17612 raft_consensus.cc:1261] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Refusing update from remote peer a62d539f6b9846359bbd85ecdcbbf327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:51:57.093305 18128 consensus_queue.cc:1035] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:51:57.093698 18125 consensus_queue.cc:1035] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:51:57.119429 17388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:51:57.119704 17388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:51:57.123428 17388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Sending DeleteTablet for 3 replicas of tablet e0a03ad3c2b641e4b27ad662e5963eaf
I20260808 15:51:57.124589 17782 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a03ad3c2b641e4b27ad662e5963eaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:51:57 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:50932
I20260808 15:51:57.124583 17987 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a03ad3c2b641e4b27ad662e5963eaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:51:57 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:60726
I20260808 15:51:57.124583 17583 tablet_service.cc:1514] Processing DeleteTablet for tablet e0a03ad3c2b641e4b27ad662e5963eaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:51:57 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:38914
I20260808 15:51:57.128123 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 17487
I20260808 15:51:57.137619 18147 tablet_replica.cc:330] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: stopping tablet replica
I20260808 15:51:57.138301 18144 tablet_replica.cc:330] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: stopping tablet replica
I20260808 15:51:57.138871 18144 raft_consensus.cc:2229] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:51:57.139487 18144 raft_consensus.cc:2258] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:51:57.140276 18147 raft_consensus.cc:2229] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:51:57.140801 18147 raft_consensus.cc:2258] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:51:57.143188 18147 ts_tablet_manager.cc:1905] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 08 15:51:57 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:51:57.153843 18144 ts_tablet_manager.cc:1905] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:51:57.155892 18143 tablet_replica.cc:330] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: stopping tablet replica
I20260808 15:51:57.156431 18143 raft_consensus.cc:2229] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:51:57.157214 18143 raft_consensus.cc:2258] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:51:57.150573 18147 ts_tablet_manager.cc:1918] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:51:57.158986 18147 log.cc:1196] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/e0a03ad3c2b641e4b27ad662e5963eaf
I20260808 15:51:57.159654 18147 ts_tablet_manager.cc:1939] T e0a03ad3c2b641e4b27ad662e5963eaf P aff13f6354df4055a769f2418f28398b: Deleting consensus metadata
I20260808 15:51:57.161953 17362 catalog_manager.cc:4841] TS aff13f6354df4055a769f2418f28398b (127.27.84.193:43453): tablet e0a03ad3c2b641e4b27ad662e5963eaf (table test-table [id=daa518193c75423290afb01d9f7df085]) successfully deleted
I20260808 15:51:57.163699 18144 ts_tablet_manager.cc:1918] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:51:57.163892 18144 log.cc:1196] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/e0a03ad3c2b641e4b27ad662e5963eaf
I20260808 15:51:57.164501 18144 ts_tablet_manager.cc:1939] T e0a03ad3c2b641e4b27ad662e5963eaf P e2d533efa1c44386a1d472cf08e91b13: Deleting consensus metadata
I20260808 15:51:57.165982 18143 ts_tablet_manager.cc:1905] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:51:57.166530 17367 catalog_manager.cc:4841] TS e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933): tablet e0a03ad3c2b641e4b27ad662e5963eaf (table test-table [id=daa518193c75423290afb01d9f7df085]) successfully deleted
I20260808 15:51:57.167387 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.129136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.27.84.193:43453)
0808 15:51:57.129384 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:57.129390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:57.135351 (+ 5961us) server_negotiation.cc:407] Connection header received
0808 15:51:57.138864 (+ 3513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:57.138874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:57.139001 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:57.139117 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:57.142102 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.143662 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.146481 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.147463 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.147591 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.155575 (+ 7984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:57.155593 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:57.155613 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:57.155669 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:57.163324 (+ 7655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:57.164435 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:57.164446 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:57.164449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:57.164637 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:57.165183 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:57.165188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:57.165191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:57.165715 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:57.166019 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:57.166722 (+ 703us) server_negotiation.cc:299] Negotiation successful
0808 15:51:57.166985 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260808 15:51:57.168725 17674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:57.174273 18143 ts_tablet_manager.cc:1918] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:51:57.174472 18143 log.cc:1196] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/e0a03ad3c2b641e4b27ad662e5963eaf
I20260808 15:51:57.175010 18143 ts_tablet_manager.cc:1939] T e0a03ad3c2b641e4b27ad662e5963eaf P a62d539f6b9846359bbd85ecdcbbf327: Deleting consensus metadata
I20260808 15:51:57.176647 17368 catalog_manager.cc:4841] TS a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194:42115): tablet e0a03ad3c2b641e4b27ad662e5963eaf (table test-table [id=daa518193c75423290afb01d9f7df085]) successfully deleted
I20260808 15:51:57.303532 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 17487
I20260808 15:51:57.329429 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 17690
Aug 08 15:51:57 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:51:57.354419 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.331180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.27.84.194:42115)
0808 15:51:57.331399 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:57.331404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:57.331797 (+ 393us) server_negotiation.cc:407] Connection header received
0808 15:51:57.331948 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:57.331955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:57.332050 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:57.332187 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:57.333438 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.334880 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.336175 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.336951 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.337094 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.346479 (+ 9385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:57.346502 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:57.346525 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:57.346583 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:57.351232 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:57.352010 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:57.352021 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:57.352025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:57.352215 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:57.352584 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:57.352590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:57.352594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:57.353138 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:57.353507 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:57.353693 (+ 186us) server_negotiation.cc:299] Negotiation successful
0808 15:51:57.353971 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:51:57.355325 17874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:57.474534 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 17690
I20260808 15:51:57.499966 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 17890
Aug 08 15:51:57 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:51:57.515468 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:57.500955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.27.84.195:44933)
0808 15:51:57.501131 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:57.501135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:57.501154 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:51:57.501362 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:57.501367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:57.501440 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:57.501537 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:57.502947 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.503788 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.504999 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:57.505388 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:57.505486 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:57.509932 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:57.509949 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:57.509967 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:57.510003 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:57.512877 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:57.513553 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:57.513561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:57.513564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:57.513704 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:57.514026 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:57.514030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:57.514032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:57.514388 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:57.514674 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:57.514911 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 15:51:57.515093 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260808 15:51:57.516146 18075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:57.607869 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 17890
I20260808 15:51:57.636448 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 17339
I20260808 15:51:57.637650 17438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:51:57.749945 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 17339
I20260808 15:51:57.778811 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43565
--webserver_interface=127.27.84.254
--webserver_port=45123
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:51:58.135746 18153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:58.136051 18153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:58.136224 18153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:58.143587 18153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:51:58.143702 18153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:58.143816 18153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:58.143918 18153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:51:58.143991 18153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:51:58.153906 18153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=45123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:58.155349 18153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:58.156778 18153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:58.184082 18161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:58.184243 18165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:58.184170 18163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:58.185386 18153 server_base.cc:1034] running on GCE node
I20260808 15:51:58.186282 18153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:58.188017 18153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:58.189240 18153 hybrid_clock.cc:648] HybridClock initialized: now 1786204318189179 us; error 46 us; skew 500 ppm
I20260808 15:51:58.189736 18153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:58 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:58.193559 18153 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:51:58.195358 18153 webserver.cc:460] Webserver started at http://127.27.84.254:45123/ using document root <none> and password file <none>
I20260808 15:51:58.196153 18153 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:58.196333 18153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:58.202721 18153 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260808 15:51:58.205878 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:58.207468 18153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:51:58.207670 18153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "74595f1069fd46f4abb46201ac4d3699"
format_stamp: "Formatted at 2026-08-08 15:51:54 on dist-test-slave-twwt"
server_key: "b164f7dc8c88a737bff707f6804088ce"
server_key_iv: "c9653c81cdcb7b1460302bea4e392368"
server_key_version: "encryptionkey@0"
I20260808 15:51:58.208806 18153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:58.226078 18153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:58.230892 18153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:58.231199 18153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:58.251623 18153 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43565
I20260808 15:51:58.251644 18260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43565 every 8 connection(s)
I20260808 15:51:58.253419 18153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:51:58.259254 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 18153
I20260808 15:51:58.260332 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:43453
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:43565
--webserver_port=36023
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:51:58.261999 18261 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:51:58.267474 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap starting.
I20260808 15:51:58.289537 18261 log.cc:824] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:58.302853 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:51:58.303423 18261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap complete.
I20260808 15:51:58.313349 18261 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:58.314229 18261 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74595f1069fd46f4abb46201ac4d3699, State: Initialized, Role: FOLLOWER
I20260808 15:51:58.314936 18261 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:58.315186 18261 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:51:58.315335 18261 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:51:58.315526 18261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:51:58.319451 18261 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:58.319993 18261 leader_election.cc:302] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74595f1069fd46f4abb46201ac4d3699; no voters:
I20260808 15:51:58.320694 18261 leader_election.cc:288] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:51:58.320845 18267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:51:58.322449 18267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 LEADER]: Becoming Leader. State: Replica: 74595f1069fd46f4abb46201ac4d3699, State: Running, Role: LEADER
I20260808 15:51:58.323055 18267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:51:58.323371 18261 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:51:58.325575 18269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "74595f1069fd46f4abb46201ac4d3699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } } }
I20260808 15:51:58.325963 18269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:58.325757 18268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74595f1069fd46f4abb46201ac4d3699. Latest consensus state: current_term: 2 leader_uuid: "74595f1069fd46f4abb46201ac4d3699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } } }
I20260808 15:51:58.326189 18268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: This master's current role is: LEADER
I20260808 15:51:58.335660 18276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:51:58.344587 18276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:51:58.346117 18276 catalog_manager.cc:1259] Loaded cluster ID: 3f7df20644db4b49a2010e8ecc9b7fce
I20260808 15:51:58.346278 18276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:51:58.350063 18276 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260808 15:51:58.368069 18276 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260808 15:51:58.376911 18276 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:51:58.378520 18276 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Loaded TSK: 0
I20260808 15:51:58.379662 18276 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:51:58.666035 18265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:58.666381 18265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:58.666549 18265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:58.674006 18265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:58.674181 18265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:58.674350 18265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:51:58.684595 18265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:43453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=36023
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:58.686039 18265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:58.687527 18265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:58.702764 18301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:58.702895 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:58.702870 18304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:58.704566 18265 server_base.cc:1034] running on GCE node
I20260808 15:51:58.705361 18265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:58.706485 18265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:58.707713 18265 hybrid_clock.cc:648] HybridClock initialized: now 1786204318707599 us; error 105 us; skew 500 ppm
I20260808 15:51:58.708089 18265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:58 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:58.711476 18265 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:51:58.713351 18265 webserver.cc:460] Webserver started at http://127.27.84.193:36023/ using document root <none> and password file <none>
I20260808 15:51:58.714177 18265 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:58.714319 18265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:58.720196 18265 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:51:58.723403 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:58.725119 18265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:51:58.725301 18265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "aff13f6354df4055a769f2418f28398b"
format_stamp: "Formatted at 2026-08-08 15:51:55 on dist-test-slave-twwt"
server_key: "18b228c1ebdbb321101ecc65c576ed50"
server_key_iv: "d9ceb652ab13fa9975afe3b6ffcfdabe"
server_key_version: "encryptionkey@0"
I20260808 15:51:58.726373 18265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:58.752010 18265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:58.756592 18265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:58.756906 18265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:58.758020 18265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:58.759505 18265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:58.759636 18265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:58.759768 18265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:58.759856 18265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:58.794381 18265 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43453
I20260808 15:51:58.794474 18482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43453 every 8 connection(s)
I20260808 15:51:58.795992 18265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:51:58.805099 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 18265
I20260808 15:51:58.805778 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:42115
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:43565
--webserver_port=33461
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:51:58 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:58.822389 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:58.799081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:45659 (local address 127.27.84.254:43565)
0808 15:51:58.799632 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:58.799644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:58.800504 (+ 860us) server_negotiation.cc:407] Connection header received
0808 15:51:58.802554 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:58.802567 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:58.802980 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:58.803288 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:58.805190 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:58.806592 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:58.809210 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:58.809875 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:58.810023 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:58.814553 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:58.814562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:58.814577 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:58.814678 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:58.817604 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:58.818388 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:58.818396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:58.818401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:58.818581 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:58.819083 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:58.819087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:58.819090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:58.819589 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:58.819790 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:58.820614 (+ 824us) server_negotiation.cc:299] Negotiation successful
0808 15:51:58.821009 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":228,"threads_started":1}
I20260808 15:51:58.822379 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:58.799419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:45659)
0808 15:51:58.800311 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:58.800350 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:58.802062 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:58.803479 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:58.803487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:58.804125 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:58.804928 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:58.804960 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:58.806832 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:58.806841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:58.809035 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:58.809054 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:58.810134 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:58.810144 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:58.810471 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:58.811281 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:58.811316 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:58.814258 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:58.817760 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:58.817769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:58.817785 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:58.818234 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:58.818750 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:58.818755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:58.818762 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:58.818956 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:58.819747 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:58.819756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:58.820101 (+ 345us) client_negotiation.cc:769] Sending connection context
0808 15:51:58.820376 (+ 275us) client_negotiation.cc:240] Negotiation successful
0808 15:51:58.820981 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":356,"threads_started":1}
I20260808 15:51:58.825426 18483 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43565
I20260808 15:51:58.825784 18483 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:58.826800 18483 heartbeater.cc:511] Master 127.27.84.254:43565 requested a full tablet report, sending...
I20260808 15:51:58.829123 18202 ts_manager.cc:194] Registered new tserver with Master: aff13f6354df4055a769f2418f28398b (127.27.84.193:43453)
I20260808 15:51:58.830865 18202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:45659
W20260808 15:51:59.166522 18487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:59.166882 18487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:59.166985 18487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:59.174638 18487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:59.174784 18487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:59.174906 18487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:51:59.185302 18487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:42115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=33461
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:59.186774 18487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:59.188208 18487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:59.201787 18495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:59.201951 18496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:59.201874 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:59.203115 18487 server_base.cc:1034] running on GCE node
I20260808 15:51:59.203759 18487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:59.204854 18487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:59.206082 18487 hybrid_clock.cc:648] HybridClock initialized: now 1786204319206028 us; error 47 us; skew 500 ppm
I20260808 15:51:59.206431 18487 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:59 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:59.209738 18487 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:51:59.211180 18487 webserver.cc:460] Webserver started at http://127.27.84.194:33461/ using document root <none> and password file <none>
I20260808 15:51:59.211891 18487 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:59.212019 18487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:59.217649 18487 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 15:51:59.220733 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:59.222199 18487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:51:59.222376 18487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "a62d539f6b9846359bbd85ecdcbbf327"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "e0e4de4c05b385a8cf6816216a8783ca"
server_key_iv: "9d3e6a621b32712475a0095273a9829e"
server_key_version: "encryptionkey@0"
I20260808 15:51:59.223549 18487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:59.249023 18487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:59.253883 18487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:59.254199 18487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:59.255378 18487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:59.257010 18487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:59.257141 18487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:59.257270 18487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:59.257366 18487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:59.297049 18487 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:42115
I20260808 15:51:59.297138 18681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:42115 every 8 connection(s)
I20260808 15:51:59.298671 18487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:51:59.300562 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 18487
I20260808 15:51:59.301282 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:44933
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:43565
--webserver_port=45041
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:51:59 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:59.323786 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.301849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:43263 (local address 127.27.84.254:43565)
0808 15:51:59.302039 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.302043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.303616 (+ 1573us) server_negotiation.cc:407] Connection header received
0808 15:51:59.305870 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.305876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.305994 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.306121 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:59.307900 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.308675 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.310455 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.311024 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.311174 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.315505 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:59.315514 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:59.315523 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:59.315576 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:59.319816 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:59.320633 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:59.320643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:59.320650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:59.320843 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:59.321427 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:59.321436 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:59.321443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:59.321919 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:59.322095 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.322867 (+ 772us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.323209 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:51:59.324666 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.302398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:43263)
0808 15:51:59.303408 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:59.303460 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:59.305547 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:59.306403 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:59.306413 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:59.307066 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:59.307702 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.307722 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.308872 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.308876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.310276 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.310290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.311296 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.311306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.311560 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.312401 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:59.312428 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:59.315244 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:59.319984 (+ 4740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:59.319992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:59.320007 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:59.320484 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:59.320998 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:59.321002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:59.321004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:59.321294 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:59.322045 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:59.322051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:59.322403 (+ 352us) client_negotiation.cc:769] Sending connection context
0808 15:51:59.322712 (+ 309us) client_negotiation.cc:240] Negotiation successful
0808 15:51:59.323203 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":448,"threads_started":1}
I20260808 15:51:59.326436 18682 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43565
I20260808 15:51:59.326807 18682 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:59.327677 18682 heartbeater.cc:511] Master 127.27.84.254:43565 requested a full tablet report, sending...
I20260808 15:51:59.329308 18202 ts_manager.cc:194] Registered new tserver with Master: a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194:42115)
I20260808 15:51:59.330217 18202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:43263
W20260808 15:51:59.677520 18685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:51:59.677882 18685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:51:59.678063 18685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:51:59.685549 18685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:51:59.685724 18685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:51:59.685899 18685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:51:59.696025 18685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:44933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=45041
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:51:59.697590 18685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:51:59.699096 18685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:51:59.712759 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:59.712816 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:51:59.713063 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:51:59.713804 18685 server_base.cc:1034] running on GCE node
I20260808 15:51:59.714449 18685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:51:59.715440 18685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:51:59.716636 18685 hybrid_clock.cc:648] HybridClock initialized: now 1786204319716571 us; error 62 us; skew 500 ppm
I20260808 15:51:59.716938 18685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:51:59 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:51:59.720196 18685 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:51:59.721702 18685 webserver.cc:460] Webserver started at http://127.27.84.195:45041/ using document root <none> and password file <none>
I20260808 15:51:59.722383 18685 fs_manager.cc:359] Metadata directory not provided
I20260808 15:51:59.722505 18685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:51:59.728523 18685 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:51:59.731467 18707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:59.732933 18685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:51:59.733119 18685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "e2d533efa1c44386a1d472cf08e91b13"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "7c22f5cb880fd45daad4c02f970b6015"
server_key_iv: "2fdc0ea629cca08fcb270534fb2bf0f1"
server_key_version: "encryptionkey@0"
I20260808 15:51:59.734148 18685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:51:59.761986 18685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:51:59.766871 18685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:51:59.767199 18685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:51:59.768308 18685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:51:59.769783 18685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:51:59.769924 18685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:59.770057 18685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:51:59.770144 18685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:51:59.804994 18685 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:44933
I20260808 15:51:59.805086 18872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:44933 every 8 connection(s)
I20260808 15:51:59.806571 18685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:51:59.808274 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 18685
Aug 08 15:51:59 dist-test-slave-twwt krb5kdc[17314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:51:59.830084 18485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.809341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.27.84.254:43565)
0808 15:51:59.809615 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.809624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.815580 (+ 5956us) server_negotiation.cc:407] Connection header received
0808 15:51:59.815749 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.815754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.815823 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.815923 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:59.817574 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.818422 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.819815 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.820332 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.820430 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.823550 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:59.823559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:59.823564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:59.823605 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:59.826874 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:59.827891 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:59.827895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:59.827897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:59.828038 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:59.828445 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:59.828450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:59.828455 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:59.828843 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:59.829120 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.829516 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.829758 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":116}
I20260808 15:51:59.832257 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.809347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:52071 (local address 127.27.84.254:43565)
0808 15:51:59.814983 (+ 5636us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.814988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.815006 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:51:59.815220 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.815226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.815327 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.815454 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:51:59.817202 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.818142 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.819890 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.820485 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.820619 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.824446 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:51:59.824454 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:51:59.824461 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:51:59.824525 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:51:59.828034 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:59.829058 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:59.829063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:59.829069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:59.829210 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:51:59.829751 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:51:59.829758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:51:59.829762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:51:59.830179 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:51:59.830328 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.831424 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.831647 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":5542,"thread_start_us":5352,"threads_started":1}
I20260808 15:51:59.833364 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.810151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:52071)
0808 15:51:59.811385 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:59.811447 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:59.814837 (+ 3390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:59.815750 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:59.815762 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:59.816412 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:51:59.817008 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.817026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.818307 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.818312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.819717 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.819755 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.820718 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.820723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.820929 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.821617 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:51:59.821640 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:51:59.824207 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:51:59.828175 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:59.828184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:59.828207 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:59.828921 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:59.829340 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:51:59.829345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:51:59.829349 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:51:59.829608 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:51:59.830426 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:51:59.830435 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:51:59.830969 (+ 534us) client_negotiation.cc:769] Sending connection context
0808 15:51:59.831339 (+ 370us) client_negotiation.cc:240] Negotiation successful
0808 15:51:59.831943 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":536,"threads_started":1}
I20260808 15:51:59.834863 18873 heartbeater.cc:347] Connected to a master server at 127.27.84.254:43565
I20260808 15:51:59.834987 18483 heartbeater.cc:503] Master 127.27.84.254:43565 was elected leader, sending a full tablet report...
I20260808 15:51:59.835210 18873 heartbeater.cc:464] Registering TS with master...
I20260808 15:51:59.836043 18873 heartbeater.cc:511] Master 127.27.84.254:43565 requested a full tablet report, sending...
I20260808 15:51:59.837584 18202 ts_manager.cc:194] Registered new tserver with Master: e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933)
I20260808 15:51:59.838497 18202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:52071
I20260808 15:51:59.845176 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:51:59.852373 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.846186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.27.84.254:43565)
0808 15:51:59.846367 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.846371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.846951 (+ 580us) server_negotiation.cc:407] Connection header received
0808 15:51:59.847105 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.847108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.847238 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.847376 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:51:59.847789 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.848525 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.850105 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.850567 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.850789 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.851296 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:51:59.851865 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:51:59.851989 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.852092 (+ 103us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.852164 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:51:59.853401 18202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:51:59.860742 18202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:51:59.895601 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.874110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.27.84.194:42115)
0808 15:51:59.887041 (+ 12931us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.887052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.887081 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:51:59.887175 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.887182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.887394 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.887589 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:59.888464 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.890008 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.892549 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.894290 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.894458 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.894723 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.894996 (+ 273us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.895163 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12771,"thread_start_us":12674,"threads_started":1}
I20260808 15:51:59.895797 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.873453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42115 (local address 127.0.0.1:50966)
0808 15:51:59.885453 (+ 12000us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:59.885480 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:59.885641 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:59.887649 (+ 2008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:59.887658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:59.887709 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:59.888248 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.888262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.890176 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.890185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.892334 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.892352 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.894511 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.894517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.894735 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.894801 (+ 66us) client_negotiation.cc:769] Sending connection context
0808 15:51:59.894948 (+ 147us) client_negotiation.cc:240] Negotiation successful
0808 15:51:59.895146 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11829,"mutex_wait_us":40,"spinlock_wait_cycles":4480,"thread_start_us":12335,"threads_started":1}
I20260808 15:51:59.897066 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.874918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.27.84.193:43453)
0808 15:51:59.884803 (+ 9885us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.884818 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.886586 (+ 1768us) server_negotiation.cc:407] Connection header received
0808 15:51:59.887024 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.887034 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.887328 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.887590 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:59.889189 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.891100 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.893749 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.895209 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.895351 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.895573 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.896549 (+ 976us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.896714 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9654,"thread_start_us":9435,"threads_started":1}
I20260808 15:51:59.897090 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.874082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43453 (local address 127.0.0.1:38946)
0808 15:51:59.886427 (+ 12345us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:59.886453 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:59.886755 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:59.887649 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:59.887658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:59.887703 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:59.888795 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.888806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.891269 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.891275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.893560 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.893574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.896035 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.896044 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.896330 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.896356 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:51:59.896490 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 15:51:59.896625 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12187,"mutex_wait_us":64,"spinlock_wait_cycles":768,"thread_start_us":8783,"threads_started":1}
I20260808 15:51:59.897137 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.872943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:44933 (local address 127.0.0.1:60744)
0808 15:51:59.883498 (+ 10555us) negotiation.cc:107] Waiting for socket to connect
0808 15:51:59.883580 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 15:51:59.883902 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:51:59.888785 (+ 4883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:51:59.888790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:51:59.888815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:51:59.889428 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.889445 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.891830 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.891844 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.894133 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:51:59.894148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.896258 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.896268 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:51:59.896466 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.896501 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:51:59.896605 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:51:59.896744 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10094,"thread_start_us":13104,"threads_started":1}
I20260808 15:51:59.898026 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:51:59.873145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.27.84.195:44933)
0808 15:51:59.888064 (+ 14919us) server_negotiation.cc:206] Beginning negotiation
0808 15:51:59.888075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:51:59.888112 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:51:59.888218 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:51:59.888226 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:51:59.888520 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:51:59.889686 (+ 1166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:51:59.890034 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.891665 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.894325 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:51:59.895827 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:51:59.895944 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:51:59.896162 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 15:51:59.897437 (+ 1275us) server_negotiation.cc:299] Negotiation successful
0808 15:51:59.897635 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14729,"thread_start_us":13397,"threads_started":1}
I20260808 15:51:59.900827 18584 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfd3c26177f4db98e3137e49069cdf0 (DEFAULT_TABLE table=test-table [id=2754af789dda446e8175aa59bdb4c684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:59.901146 18386 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfd3c26177f4db98e3137e49069cdf0 (DEFAULT_TABLE table=test-table [id=2754af789dda446e8175aa59bdb4c684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:59.902596 18584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfd3c26177f4db98e3137e49069cdf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:59.902648 18386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfd3c26177f4db98e3137e49069cdf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:59.903003 18782 tablet_service.cc:1467] Processing CreateTablet for tablet 3bfd3c26177f4db98e3137e49069cdf0 (DEFAULT_TABLE table=test-table [id=2754af789dda446e8175aa59bdb4c684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:51:59.904394 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bfd3c26177f4db98e3137e49069cdf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:51:59.917152 18901 tablet_bootstrap.cc:492] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Bootstrap starting.
I20260808 15:51:59.919726 18902 tablet_bootstrap.cc:492] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Bootstrap starting.
I20260808 15:51:59.921470 18903 tablet_bootstrap.cc:492] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Bootstrap starting.
I20260808 15:51:59.923624 18901 tablet_bootstrap.cc:654] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:59.924104 18902 tablet_bootstrap.cc:654] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:59.924999 18903 tablet_bootstrap.cc:654] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Neither blocks nor log segments found. Creating new log.
I20260808 15:51:59.925161 18901 log.cc:824] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:59.925526 18902 log.cc:824] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:59.926210 18903 log.cc:824] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Log is configured to *not* fsync() on all Append() calls
I20260808 15:51:59.929167 18901 tablet_bootstrap.cc:492] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: No bootstrap required, opened a new log
I20260808 15:51:59.929167 18902 tablet_bootstrap.cc:492] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: No bootstrap required, opened a new log
I20260808 15:51:59.929481 18902 ts_tablet_manager.cc:1397] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260808 15:51:59.929481 18901 ts_tablet_manager.cc:1397] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260808 15:51:59.929790 18903 tablet_bootstrap.cc:492] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: No bootstrap required, opened a new log
I20260808 15:51:59.930120 18903 ts_tablet_manager.cc:1397] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:51:59.938694 18903 raft_consensus.cc:348] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:51:59.939144 18903 raft_consensus.cc:374] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:59.939248 18903 raft_consensus.cc:729] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d533efa1c44386a1d472cf08e91b13, State: Initialized, Role: FOLLOWER
I20260808 15:51:59.939900 18903 consensus_queue.cc:260] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:51:59.941092 18873 heartbeater.cc:503] Master 127.27.84.254:43565 was elected leader, sending a full tablet report...
I20260808 15:51:59.941653 18903 ts_tablet_manager.cc:1428] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260808 15:51:59.941736 18902 raft_consensus.cc:348] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:51:59.942193 18902 raft_consensus.cc:374] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:59.942312 18902 raft_consensus.cc:729] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62d539f6b9846359bbd85ecdcbbf327, State: Initialized, Role: FOLLOWER
I20260808 15:51:59.945031 18901 raft_consensus.cc:348] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:51:59.945472 18902 consensus_queue.cc:260] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:51:59.945537 18901 raft_consensus.cc:374] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:51:59.946511 18901 raft_consensus.cc:729] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aff13f6354df4055a769f2418f28398b, State: Initialized, Role: FOLLOWER
I20260808 15:51:59.946683 18682 heartbeater.cc:503] Master 127.27.84.254:43565 was elected leader, sending a full tablet report...
I20260808 15:51:59.947252 18901 consensus_queue.cc:260] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:51:59.948596 18902 ts_tablet_manager.cc:1428] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Time spent starting tablet: real 0.019s user 0.011s sys 0.002s
I20260808 15:51:59.950335 18901 ts_tablet_manager.cc:1428] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Time spent starting tablet: real 0.021s user 0.017s sys 0.000s
W20260808 15:52:00.048533 18484 tablet.cc:2369] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:52:00.050871 18683 tablet.cc:2369] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:52:00.058264 18874 tablet.cc:2369] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:52:00.311214 18911 raft_consensus.cc:482] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:52:00.311551 18911 raft_consensus.cc:504] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:52:00.313498 18911 leader_election.cc:288] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933), a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194:42115)
I20260808 15:52:00.320235 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.313841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:32903 (local address 127.27.84.195:44933)
0808 15:52:00.314084 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:00.314091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:00.314886 (+ 795us) server_negotiation.cc:407] Connection header received
0808 15:52:00.315162 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:00.315167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:00.315230 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:00.315331 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:00.315918 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.316618 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.318416 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.319259 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.319448 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.319527 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:00.319804 (+ 277us) server_negotiation.cc:299] Negotiation successful
0808 15:52:00.319940 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 15:52:00.320361 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.313793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:44933 (local address 127.27.84.193:32903)
0808 15:52:00.314755 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:00.314777 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:00.314996 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:00.315357 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:00.315361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:00.315389 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:00.315763 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:00.315773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.316752 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.316760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:00.318249 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:00.318262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.319485 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.319491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:00.319641 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.319666 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:52:00.319776 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 15:52:00.319906 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":1,"thread_start_us":562,"threads_started":1}
I20260808 15:52:00.321070 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.314049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:40821 (local address 127.27.84.194:42115)
0808 15:52:00.314374 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:00.314384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:00.315195 (+ 811us) server_negotiation.cc:407] Connection header received
0808 15:52:00.315430 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:00.315435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:00.315495 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:00.315607 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:00.316247 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.317037 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.318979 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.319848 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.320245 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.320328 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:00.320667 (+ 339us) server_negotiation.cc:299] Negotiation successful
0808 15:52:00.320830 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260808 15:52:00.321092 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.314127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42115 (local address 127.27.84.193:40821)
0808 15:52:00.315077 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:00.315093 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:00.315275 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:00.315630 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:00.315634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:00.315662 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:00.316045 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:00.316055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.317212 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.317217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:00.318816 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:00.318829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.320299 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.320305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:00.320485 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.320517 (+ 32us) client_negotiation.cc:769] Sending connection context
0808 15:52:00.320640 (+ 123us) client_negotiation.cc:240] Negotiation successful
0808 15:52:00.320765 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":566,"threads_started":1}
I20260808 15:52:00.321545 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd3c26177f4db98e3137e49069cdf0" candidate_uuid: "aff13f6354df4055a769f2418f28398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d533efa1c44386a1d472cf08e91b13" is_pre_election: true
I20260808 15:52:00.322049 18808 raft_consensus.cc:2454] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff13f6354df4055a769f2418f28398b in term 0.
I20260808 15:52:00.322121 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd3c26177f4db98e3137e49069cdf0" candidate_uuid: "aff13f6354df4055a769f2418f28398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62d539f6b9846359bbd85ecdcbbf327" is_pre_election: true
I20260808 15:52:00.322770 18613 raft_consensus.cc:2454] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aff13f6354df4055a769f2418f28398b in term 0.
I20260808 15:52:00.322782 18318 leader_election.cc:302] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aff13f6354df4055a769f2418f28398b, e2d533efa1c44386a1d472cf08e91b13; no voters:
I20260808 15:52:00.323223 18911 raft_consensus.cc:2781] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:52:00.323416 18911 raft_consensus.cc:482] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:52:00.323532 18911 raft_consensus.cc:3037] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:00.325868 18911 raft_consensus.cc:504] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:52:00.326478 18911 leader_election.cc:288] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [CANDIDATE]: Term 1 election: Requested vote from peers e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933), a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194:42115)
I20260808 15:52:00.326985 18808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd3c26177f4db98e3137e49069cdf0" candidate_uuid: "aff13f6354df4055a769f2418f28398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2d533efa1c44386a1d472cf08e91b13"
I20260808 15:52:00.327021 18613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bfd3c26177f4db98e3137e49069cdf0" candidate_uuid: "aff13f6354df4055a769f2418f28398b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62d539f6b9846359bbd85ecdcbbf327"
I20260808 15:52:00.327186 18808 raft_consensus.cc:3037] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:00.327236 18613 raft_consensus.cc:3037] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:00.329628 18808 raft_consensus.cc:2454] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff13f6354df4055a769f2418f28398b in term 1.
I20260808 15:52:00.329931 18613 raft_consensus.cc:2454] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aff13f6354df4055a769f2418f28398b in term 1.
I20260808 15:52:00.330341 18318 leader_election.cc:302] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aff13f6354df4055a769f2418f28398b, e2d533efa1c44386a1d472cf08e91b13; no voters:
I20260808 15:52:00.330680 18911 raft_consensus.cc:2781] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:52:00.331210 18911 raft_consensus.cc:686] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 1 LEADER]: Becoming Leader. State: Replica: aff13f6354df4055a769f2418f28398b, State: Running, Role: LEADER
I20260808 15:52:00.331725 18911 consensus_queue.cc:237] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } }
I20260808 15:52:00.337076 18201 catalog_manager.cc:5495] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b reported cstate change: term changed from 0 to 1, leader changed from <none> to aff13f6354df4055a769f2418f28398b (127.27.84.193). New cstate: current_term: 1 leader_uuid: "aff13f6354df4055a769f2418f28398b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aff13f6354df4055a769f2418f28398b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43453 } health_report { overall_health: HEALTHY } } }
I20260808 15:52:00.460275 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.454487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.27.84.193:43453)
0808 15:52:00.455018 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:00.455023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:00.455042 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:52:00.455241 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:00.455245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:00.455327 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:00.455471 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:52:00.455902 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.456615 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.458052 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.458462 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.458560 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.459143 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:52:00.459675 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:52:00.459849 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:00.459976 (+ 127us) server_negotiation.cc:299] Negotiation successful
0808 15:52:00.460049 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
I20260808 15:52:00.468605 18808 raft_consensus.cc:1261] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Refusing update from remote peer aff13f6354df4055a769f2418f28398b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:52:00.468600 18613 raft_consensus.cc:1261] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Refusing update from remote peer aff13f6354df4055a769f2418f28398b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:52:00.469645 18916 consensus_queue.cc:1035] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2d533efa1c44386a1d472cf08e91b13" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:52:00.469980 18911 consensus_queue.cc:1035] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62d539f6b9846359bbd85ecdcbbf327" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:52:00.483387 18923 mvcc.cc:205] Tried to move back new op lower bound from 7316292896628826112 to 7316292896082526208. Current Snapshot: MvccSnapshot[applied={T|T < 7316292896628826112}]
I20260808 15:52:00.484351 18927 mvcc.cc:205] Tried to move back new op lower bound from 7316292896628826112 to 7316292896082526208. Current Snapshot: MvccSnapshot[applied={T|T < 7316292896628826112}]
I20260808 15:52:00.499532 18201 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:52:00.499864 18201 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:52:00.503603 18201 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Sending DeleteTablet for 3 replicas of tablet 3bfd3c26177f4db98e3137e49069cdf0
I20260808 15:52:00.504937 18584 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfd3c26177f4db98e3137e49069cdf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:00 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:50966
I20260808 15:52:00.504928 18782 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfd3c26177f4db98e3137e49069cdf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:00 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:60744
I20260808 15:52:00.505194 18386 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bfd3c26177f4db98e3137e49069cdf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:00 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:38946
I20260808 15:52:00.506098 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 18265
I20260808 15:52:00.523283 18941 tablet_replica.cc:330] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: stopping tablet replica
I20260808 15:52:00.523890 18941 raft_consensus.cc:2229] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:52:00.524442 18939 tablet_replica.cc:330] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: stopping tablet replica
I20260808 15:52:00.524703 18941 raft_consensus.cc:2258] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:00.526990 18941 ts_tablet_manager.cc:1905] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:00.527855 18940 tablet_replica.cc:330] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: stopping tablet replica
I20260808 15:52:00.528311 18940 raft_consensus.cc:2229] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:52:00.528841 18940 raft_consensus.cc:2258] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:00.532222 18939 raft_consensus.cc:2229] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:52:00.532869 18939 raft_consensus.cc:2258] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:00.533066 18940 ts_tablet_manager.cc:1905] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:00.534886 18939 ts_tablet_manager.cc:1905] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:00.539829 18941 ts_tablet_manager.cc:1918] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:00.540038 18941 log.cc:1196] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/3bfd3c26177f4db98e3137e49069cdf0
I20260808 15:52:00.540349 18940 ts_tablet_manager.cc:1918] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:00.540524 18940 log.cc:1196] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/3bfd3c26177f4db98e3137e49069cdf0
I20260808 15:52:00.540815 18941 ts_tablet_manager.cc:1939] T 3bfd3c26177f4db98e3137e49069cdf0 P aff13f6354df4055a769f2418f28398b: Deleting consensus metadata
I20260808 15:52:00.541136 18940 ts_tablet_manager.cc:1939] T 3bfd3c26177f4db98e3137e49069cdf0 P e2d533efa1c44386a1d472cf08e91b13: Deleting consensus metadata
I20260808 15:52:00.542860 18179 catalog_manager.cc:4841] TS aff13f6354df4055a769f2418f28398b (127.27.84.193:43453): tablet 3bfd3c26177f4db98e3137e49069cdf0 (table test-table [id=2754af789dda446e8175aa59bdb4c684]) successfully deleted
I20260808 15:52:00.542881 18182 catalog_manager.cc:4841] TS e2d533efa1c44386a1d472cf08e91b13 (127.27.84.195:44933): tablet 3bfd3c26177f4db98e3137e49069cdf0 (table test-table [id=2754af789dda446e8175aa59bdb4c684]) successfully deleted
I20260808 15:52:00.546368 18939 ts_tablet_manager.cc:1918] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:00.546402 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.507048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.27.84.193:43453)
0808 15:52:00.507235 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:00.507244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:00.523685 (+ 16441us) server_negotiation.cc:407] Connection header received
0808 15:52:00.523973 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:00.523983 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:00.524111 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:00.524273 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:00.526934 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.528383 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.531249 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.531805 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.531959 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.536285 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:00.536302 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:00.536322 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:00.536393 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:00.541933 (+ 5540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:00.543207 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:00.543217 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:00.543222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:00.543463 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:00.544057 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:00.544063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:00.544067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:00.544585 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:00.544924 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:00.545658 (+ 734us) server_negotiation.cc:299] Negotiation successful
0808 15:52:00.545952 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260808 15:52:00.546568 18939 log.cc:1196] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/3bfd3c26177f4db98e3137e49069cdf0
I20260808 15:52:00.547091 18939 ts_tablet_manager.cc:1939] T 3bfd3c26177f4db98e3137e49069cdf0 P a62d539f6b9846359bbd85ecdcbbf327: Deleting consensus metadata
I20260808 15:52:00.548173 18473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:00.549140 18183 catalog_manager.cc:4841] TS a62d539f6b9846359bbd85ecdcbbf327 (127.27.84.194:42115): tablet 3bfd3c26177f4db98e3137e49069cdf0 (table test-table [id=2754af789dda446e8175aa59bdb4c684]) successfully deleted
I20260808 15:52:00.655126 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 18265
I20260808 15:52:00.689008 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 18487
I20260808 15:52:00.709220 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.689914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.27.84.194:42115)
0808 15:52:00.690107 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:00.690112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:00.690131 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:52:00.690221 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:00.690225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:00.690297 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:00.690383 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:00.691850 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.692582 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.693954 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.694395 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.694511 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.702432 (+ 7921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:00.702450 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:00.702468 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:00.702511 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:00.705827 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:00.706865 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:00.706874 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:00.706877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:00.707038 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:00.707467 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:00.707471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:00.707473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:00.708016 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:00.708312 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:00.708649 (+ 337us) server_negotiation.cc:299] Negotiation successful
0808 15:52:00.708952 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:52:00.710011 18674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:00.803012 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 18487
I20260808 15:52:00.829972 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 18685
I20260808 15:52:00.845180 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:00.830974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.27.84.195:44933)
0808 15:52:00.831533 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:00.831537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:00.831554 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:52:00.831613 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:00.831618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:00.831686 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:00.831826 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:00.833388 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.834206 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.835617 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:00.836050 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:00.836242 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:00.839195 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:00.839211 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:00.839229 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:00.839278 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:00.842091 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:00.842867 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:00.842881 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:00.842884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:00.843035 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:00.843456 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:00.843463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:00.843471 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:00.843846 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:00.844183 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:00.844551 (+ 368us) server_negotiation.cc:299] Negotiation successful
0808 15:52:00.844802 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":292,"threads_started":1}
I20260808 15:52:00.845995 18868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:00.937553 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 18685
I20260808 15:52:00.959981 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 18153
I20260808 15:52:00.961215 18252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:01.077354 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 18153
I20260808 15:52:01.111968 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43565
--webserver_interface=127.27.84.254
--webserver_port=45123
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:01.478500 18950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:01.478960 18950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:01.479125 18950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:01.486716 18950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:52:01.486853 18950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:01.486937 18950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:01.487038 18950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:52:01.487114 18950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:52:01.497455 18950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=45123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:01.498970 18950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:01.500491 18950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:01.512709 18956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:01.512765 18957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:01.512966 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:01.513746 18950 server_base.cc:1034] running on GCE node
I20260808 15:52:01.514581 18950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:01.516120 18950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:01.517373 18950 hybrid_clock.cc:648] HybridClock initialized: now 1786204321517315 us; error 66 us; skew 500 ppm
I20260808 15:52:01.517756 18950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:01 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:01.521216 18950 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:52:01.522884 18950 webserver.cc:460] Webserver started at http://127.27.84.254:45123/ using document root <none> and password file <none>
I20260808 15:52:01.523561 18950 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:01.523669 18950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:01.529191 18950 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260808 15:52:01.534078 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:01.535502 18950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:52:01.535640 18950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "74595f1069fd46f4abb46201ac4d3699"
format_stamp: "Formatted at 2026-08-08 15:51:54 on dist-test-slave-twwt"
server_key: "b164f7dc8c88a737bff707f6804088ce"
server_key_iv: "c9653c81cdcb7b1460302bea4e392368"
server_key_version: "encryptionkey@0"
I20260808 15:52:01.536593 18950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:01.569285 18950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:01.574014 18950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:01.574263 18950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:01.594157 18950 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43565
I20260808 15:52:01.594228 19049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43565 every 8 connection(s)
I20260808 15:52:01.596083 18950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:52:01.603011 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 18950
I20260808 15:52:01.603616 19051 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:52:01.603965 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:43453
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:43565
--webserver_port=36023
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:52:01.607154 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap starting.
I20260808 15:52:01.631986 19051 log.cc:824] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:01.655120 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:52:01.655840 19051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699: Bootstrap complete.
I20260808 15:52:01.669371 19051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:52:01.670852 19051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74595f1069fd46f4abb46201ac4d3699, State: Initialized, Role: FOLLOWER
I20260808 15:52:01.671690 19051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:52:01.672019 19051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:52:01.672153 19051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:52:01.672349 19051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 2 FOLLOWER]: Advancing to term 3
I20260808 15:52:01.678467 19051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:52:01.679157 19051 leader_election.cc:302] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74595f1069fd46f4abb46201ac4d3699; no voters:
I20260808 15:52:01.679888 19051 leader_election.cc:288] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 15:52:01.680068 19055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 15:52:01.681339 19055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [term 3 LEADER]: Becoming Leader. State: Replica: 74595f1069fd46f4abb46201ac4d3699, State: Running, Role: LEADER
I20260808 15:52:01.681929 19055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } }
I20260808 15:52:01.682456 19051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:52:01.684640 19058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74595f1069fd46f4abb46201ac4d3699. Latest consensus state: current_term: 3 leader_uuid: "74595f1069fd46f4abb46201ac4d3699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } } }
I20260808 15:52:01.685096 19058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: This master's current role is: LEADER
I20260808 15:52:01.685724 19057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "74595f1069fd46f4abb46201ac4d3699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74595f1069fd46f4abb46201ac4d3699" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43565 } } }
I20260808 15:52:01.686051 19057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74595f1069fd46f4abb46201ac4d3699 [sys.catalog]: This master's current role is: LEADER
I20260808 15:52:01.693470 19067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:52:01.702951 19067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:52:01.705055 19067 catalog_manager.cc:1259] Loaded cluster ID: 3f7df20644db4b49a2010e8ecc9b7fce
I20260808 15:52:01.705179 19067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260808 15:52:01.708077 19067 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2752d039ad google::LogMessage::Fail() at ??:0
@ 0x7f2752d0688a google::LogMessage::SendToLog() at ??:0
@ 0x7f2752d034de google::LogMessage::Flush() at ??:0
@ 0x7f2752d04a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2768736512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2768733976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f27687bb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f27687bb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f275594f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f275595291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2755952799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f275592deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2758c076db start_thread at ??:0
@ 0x7f2751b9d61f clone at ??:0
W20260808 15:52:02.032650 19053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:02.033073 19053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:02.033262 19053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:02.041013 19053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:02.041203 19053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:02.041386 19053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:52:02.051924 19053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:43453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=36023
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:02.053433 19053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:02.054944 19053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:02.068918 19114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:02.068804 19118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:02.068885 19115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:02.070098 19053 server_base.cc:1034] running on GCE node
I20260808 15:52:02.070796 19053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:02.071834 19053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:02.073077 19053 hybrid_clock.cc:648] HybridClock initialized: now 1786204322073022 us; error 49 us; skew 500 ppm
I20260808 15:52:02.073408 19053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:02 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:02.076700 19053 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:52:02.078246 19053 webserver.cc:460] Webserver started at http://127.27.84.193:36023/ using document root <none> and password file <none>
I20260808 15:52:02.079035 19053 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:02.079188 19053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:02.084779 19053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 15:52:02.087790 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:02.089217 19053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:52:02.089423 19053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "aff13f6354df4055a769f2418f28398b"
format_stamp: "Formatted at 2026-08-08 15:51:55 on dist-test-slave-twwt"
server_key: "18b228c1ebdbb321101ecc65c576ed50"
server_key_iv: "d9ceb652ab13fa9975afe3b6ffcfdabe"
server_key_version: "encryptionkey@0"
I20260808 15:52:02.090454 19053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:02.107861 19053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:02.112366 19053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:02.112668 19053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:02.113725 19053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:02.115185 19053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:02.115316 19053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:02.115458 19053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:02.115561 19053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:02.151140 19053 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43453
I20260808 15:52:02.151233 19294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43453 every 8 connection(s)
I20260808 15:52:02.152670 19053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:52:02.157832 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:02.155955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:44869)
0808 15:52:02.156712 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:02.156963 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":292,"threads_started":1}
W20260808 15:52:02.158517 19295 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:43565 (0 consecutive failures): Network error: Failed to ping master at 127.27.84.254:43565: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
I20260808 15:52:02.159374 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:02.159046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:46459)
0808 15:52:02.159188 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:02.159264 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 15:52:02.160255 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:02.159861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:55087)
0808 15:52:02.160030 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:02.160113 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260808 15:52:02.160506 19295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:52:02.162319 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 19053
I20260808 15:52:02.163127 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:42115
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:43565
--webserver_port=33461
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:02.524639 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:02.524940 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:02.525038 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:02.532330 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:02.532451 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:02.532570 19298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:52:02.542479 19298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:42115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=33461
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:02.543871 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:02.545257 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:02.558813 19305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:02.558861 19307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:02.558775 19309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:02.560386 19298 server_base.cc:1034] running on GCE node
I20260808 15:52:02.561091 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:02.562103 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:02.563342 19298 hybrid_clock.cc:648] HybridClock initialized: now 1786204322563285 us; error 48 us; skew 500 ppm
I20260808 15:52:02.563680 19298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:02 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:02.567360 19298 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:52:02.569087 19298 webserver.cc:460] Webserver started at http://127.27.84.194:33461/ using document root <none> and password file <none>
I20260808 15:52:02.569893 19298 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:02.570073 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:02.575943 19298 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260808 15:52:02.578855 19319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:02.580238 19298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:52:02.580428 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "a62d539f6b9846359bbd85ecdcbbf327"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "e0e4de4c05b385a8cf6816216a8783ca"
server_key_iv: "9d3e6a621b32712475a0095273a9829e"
server_key_version: "encryptionkey@0"
I20260808 15:52:02.581475 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:02.598932 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:02.603395 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:02.603698 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:02.604740 19298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:02.606161 19298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:02.606298 19298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:02.606427 19298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:02.606520 19298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:02.641928 19298 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:42115
I20260808 15:52:02.642015 19477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:42115 every 8 connection(s)
I20260808 15:52:02.643509 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:52:02.645345 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 19298
I20260808 15:52:02.646080 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:44933
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:43565
--webserver_port=45041
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:52:02.649348 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:02.646951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:33315)
0808 15:52:02.647868 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:02.648145 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":312,"threads_started":1}
W20260808 15:52:02.650146 19480 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:43565 (0 consecutive failures): Network error: Failed to ping master at 127.27.84.254:43565: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
I20260808 15:52:02.651227 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:02.650760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:47533)
0808 15:52:02.650959 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:02.651060 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:52:02.652273 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:02.651814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:58563)
0808 15:52:02.652002 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:02.652106 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:52:02.652566 19480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:52:02.997560 19483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:02.997906 19483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:02.998093 19483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:03.005399 19483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:03.005569 19483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:03.005767 19483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:52:03.015939 19483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:44933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=45041
--tserver_master_addrs=127.27.84.254:43565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:03.017409 19483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:03.018910 19483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:03.033587 19493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:03.033552 19496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:03.033635 19494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:03.039732 19483 server_base.cc:1034] running on GCE node
I20260808 15:52:03.040612 19483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:03.041743 19483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:03.042990 19483 hybrid_clock.cc:648] HybridClock initialized: now 1786204323042918 us; error 70 us; skew 500 ppm
I20260808 15:52:03.043359 19483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:03 dist-test-slave-twwt krb5kdc[17314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204323, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:03.047066 19483 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:52:03.048698 19483 webserver.cc:460] Webserver started at http://127.27.84.195:45041/ using document root <none> and password file <none>
I20260808 15:52:03.049461 19483 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:03.049576 19483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:03.055595 19483 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:52:03.058598 19503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:03.060133 19483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:52:03.060284 19483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "e2d533efa1c44386a1d472cf08e91b13"
format_stamp: "Formatted at 2026-08-08 15:51:56 on dist-test-slave-twwt"
server_key: "7c22f5cb880fd45daad4c02f970b6015"
server_key_iv: "2fdc0ea629cca08fcb270534fb2bf0f1"
server_key_version: "encryptionkey@0"
I20260808 15:52:03.061345 19483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:03.080686 19483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:03.085507 19483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:03.085754 19483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:03.086910 19483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:03.088255 19483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:03.088347 19483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:03.088433 19483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:03.088490 19483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:03.123713 19483 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:44933
I20260808 15:52:03.123803 19679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:44933 every 8 connection(s)
I20260808 15:52:03.125249 19483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:52:03.128458 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 19483
I20260808 15:52:03.130911 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:03.128467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:49805)
0808 15:52:03.129426 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:03.129707 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":381,"threads_started":1}
W20260808 15:52:03.131769 19680 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:43565 (0 consecutive failures): Network error: Failed to ping master at 127.27.84.254:43565: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
I20260808 15:52:03.132665 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:03.132326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:36871)
0808 15:52:03.132482 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:03.132555 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 15:52:03.133569 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:03.133276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:48817)
0808 15:52:03.133395 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:03.133466 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260808 15:52:03.133870 19680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:52:03.161978 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:03.161142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:50973)
0808 15:52:03.161619 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:03.161742 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":234,"threads_started":1}
I20260808 15:52:03.654059 19692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:03.653153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:40897)
0808 15:52:03.653676 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:03.653803 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":245,"threads_started":1}
I20260808 15:52:04.135313 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:04.134398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:55495)
0808 15:52:04.134958 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:04.135068 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":316,"threads_started":1}
I20260808 15:52:04.163743 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:04.162998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:55201)
0808 15:52:04.163477 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:04.163570 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":237,"threads_started":1}
I20260808 15:52:04.655961 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:04.655116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:33835)
0808 15:52:04.655649 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:04.655737 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":271,"threads_started":1}
I20260808 15:52:05.137042 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:05.136286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:43345)
0808 15:52:05.136773 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:05.136864 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":243,"threads_started":1}
I20260808 15:52:05.165688 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:05.164846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:54043)
0808 15:52:05.165374 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:05.165470 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":264,"threads_started":1}
I20260808 15:52:05.657733 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:05.657006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:48613)
0808 15:52:05.657464 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:05.657552 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":234,"threads_started":1}
I20260808 15:52:06.138967 19704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:06.138120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:53013)
0808 15:52:06.138662 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:06.138771 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":257,"threads_started":1}
I20260808 15:52:06.167470 19706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:06.166752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:42493)
0808 15:52:06.167188 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:06.167269 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260808 15:52:06.659447 19707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:06.658680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:57587)
0808 15:52:06.659158 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:06.659240 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260808 15:52:07.140645 19709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:07.139909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:54911)
0808 15:52:07.140369 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:07.140455 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":232,"threads_started":1}
I20260808 15:52:07.169139 19710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:07.168402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:57229)
0808 15:52:07.168881 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:07.168965 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":245,"threads_started":1}
I20260808 15:52:07.661113 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:07.660354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:39973)
0808 15:52:07.660854 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:07.660938 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260808 15:52:08.142369 19712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:08.141605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:37495)
0808 15:52:08.142105 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:08.142190 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":271,"threads_started":1}
I20260808 15:52:08.171176 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:08.170309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:55829)
0808 15:52:08.170906 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:08.171003 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":358,"threads_started":1}
I20260808 15:52:08.663059 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:08.662174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:35095)
0808 15:52:08.662786 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:08.662886 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":310,"threads_started":1}
I20260808 15:52:09.144174 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:09.143394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:50577)
0808 15:52:09.143893 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:09.143999 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":252,"threads_started":1}
I20260808 15:52:09.172894 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:09.172093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:43501)
0808 15:52:09.172567 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:09.172651 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":210,"threads_started":1}
I20260808 15:52:09.664974 19721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:09.664098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:60281)
0808 15:52:09.664657 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:09.664742 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":242,"threads_started":1}
I20260808 15:52:10.146082 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:10.145262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:54733)
0808 15:52:10.145783 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:10.145867 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":246,"threads_started":1}
I20260808 15:52:10.174712 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:10.173884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:56121)
0808 15:52:10.174363 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:10.174467 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":251,"threads_started":1}
I20260808 15:52:10.667038 19726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:10.666129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:52295)
0808 15:52:10.666765 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:10.666861 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":254,"threads_started":1}
I20260808 15:52:11.148075 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:11.147282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:44903)
0808 15:52:11.147803 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:11.147894 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":274,"threads_started":1}
I20260808 15:52:11.176550 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:11.175614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:36753)
0808 15:52:11.176240 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:11.176324 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":358,"threads_started":1}
I20260808 15:52:11.668885 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:11.668081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:52341)
0808 15:52:11.668616 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:11.668703 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260808 15:52:12.149997 19732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:12.149169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:58715)
0808 15:52:12.149702 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:12.149808 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":260,"threads_started":1}
I20260808 15:52:12.178296 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:12.177531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:49885)
0808 15:52:12.178045 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:12.178127 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":305,"threads_started":1}
I20260808 15:52:12.670830 19735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:12.670021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:51345)
0808 15:52:12.670510 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:12.670600 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":265,"threads_started":1}
I20260808 15:52:13.152040 19736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:13.151147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:42985)
0808 15:52:13.151681 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:13.151836 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":257,"threads_started":1}
I20260808 15:52:13.180107 19737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:13.179322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:47595)
0808 15:52:13.179819 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:13.179919 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260808 15:52:13.672596 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:13.671826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:52677)
0808 15:52:13.672309 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:13.672406 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":269,"threads_started":1}
I20260808 15:52:14.153903 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:14.153033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:41013)
0808 15:52:14.153566 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:14.153686 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":264,"threads_started":1}
I20260808 15:52:14.182041 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:14.181205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:50869)
0808 15:52:14.181779 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:14.181870 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":307,"threads_started":1}
I20260808 15:52:14.674525 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:14.673616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:48141)
0808 15:52:14.674172 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:14.674288 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
I20260808 15:52:15.155972 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:15.155035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:57937)
0808 15:52:15.155620 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:15.155724 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20260808 15:52:15.183964 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:15.183065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:41885)
0808 15:52:15.183642 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:15.183748 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":299,"threads_started":1}
I20260808 15:52:15.676421 19745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:15.675562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:57377)
0808 15:52:15.676137 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:15.676231 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":312,"threads_started":1}
I20260808 15:52:16.157665 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:16.156913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:54963)
0808 15:52:16.157389 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:16.157480 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":226,"threads_started":1}
I20260808 15:52:16.185735 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:16.184994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:51059)
0808 15:52:16.185460 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:16.185551 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":218,"threads_started":1}
I20260808 15:52:16.678130 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:16.677379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:48201)
0808 15:52:16.677848 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:16.677946 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20260808 15:52:17.159485 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:17.158726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:57613)
0808 15:52:17.159185 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:17.159277 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
I20260808 15:52:17.187840 19755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:17.187041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.193:50685)
0808 15:52:17.187491 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:17.187596 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260808 15:52:17.680074 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:17.679324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.194:57417)
0808 15:52:17.679753 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:17.679840 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":216,"threads_started":1}
I20260808 15:52:18.131610 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 19053
I20260808 15:52:18.151419 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:18.132918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.27.84.193:43453)
0808 15:52:18.133525 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:18.133539 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:18.133576 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:52:18.135826 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:18.135836 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:18.136283 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:18.136652 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:18.138320 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:18.139730 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:18.141063 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:18.141753 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:18.141979 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:18.144825 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:18.144833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:18.144850 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:18.144894 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:18.147858 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:18.148579 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:18.148586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:18.148593 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:18.148877 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:18.149284 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:18.149289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:18.149293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:18.149829 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:18.150162 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:18.150650 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 15:52:18.151120 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":296,"threads_started":1}
I20260808 15:52:18.152930 19288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:18.161538 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:18.160759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:43565 (local address 127.27.84.195:59267)
0808 15:52:18.161247 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:18.161340 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260808 15:52:18.240093 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 19053
I20260808 15:52:18.269814 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 19298
I20260808 15:52:18.288432 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:18.270888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.27.84.194:42115)
0808 15:52:18.271514 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:18.271530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:18.271569 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:52:18.273246 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:18.273258 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:18.273706 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:18.274088 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:18.275607 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:18.276892 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:18.278185 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:18.278963 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:18.279172 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:18.281917 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:18.281925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:18.281941 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:18.281981 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:18.285003 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:18.285737 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:18.285751 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:18.285757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:18.285940 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:18.286321 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:18.286326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:18.286330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:18.286934 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:18.287270 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:18.287702 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 15:52:18.288125 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":285,"threads_started":1}
I20260808 15:52:18.289853 19471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:18.375319 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 19298
I20260808 15:52:18.405758 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 19483
I20260808 15:52:18.424974 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:18.406888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.27.84.195:44933)
0808 15:52:18.407554 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:18.407569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:18.407613 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:52:18.409282 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:18.409292 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:18.409754 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:18.410056 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:18.411822 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:18.413163 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:18.414576 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:18.415630 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:18.415791 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:18.418563 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:18.418571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:18.418590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:18.418695 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:18.421648 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:18.422374 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:18.422382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:18.422389 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:18.422567 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:18.423054 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:18.423057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:18.423059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:18.423603 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:18.423845 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:18.424183 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 15:52:18.424628 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":297,"threads_started":1}
I20260808 15:52:18.426445 19674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:18.511667 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 19483
2026-08-08T15:52:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25611 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:52:18 dist-test-slave-twwt krb5kdc[19772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:52:18 dist-test-slave-twwt krb5kdc[19772](info): set up 2 sockets
Aug 08 15:52:18 dist-test-slave-twwt krb5kdc[19772](info): commencing operation
krb5kdc: starting...
W20260808 15:52:19.867635 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:52:20 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:52:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:52:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:52:20.197868 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41247
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:20.555924 19795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:20.556212 19795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:20.556304 19795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:20.563742 19795 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:52:20.563853 19795 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:52:20.563891 19795 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:52:20.563956 19795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:52:20.564009 19795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:20.564079 19795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:20.564126 19795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:52:20.564174 19795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:52:20.574139 19795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:41247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:20.575426 19795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:20.576732 19795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:20.589114 19815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:20.589223 19803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:20.589217 19805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:20.590477 19795 server_base.cc:1034] running on GCE node
I20260808 15:52:20.591322 19795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:20.592690 19795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:20.593935 19795 hybrid_clock.cc:648] HybridClock initialized: now 1786204340593873 us; error 25 us; skew 500 ppm
I20260808 15:52:20.594310 19795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:20 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204340, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:20.603613 19795 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:52:20.605250 19795 webserver.cc:460] Webserver started at http://127.27.84.254:40731/ using document root <none> and password file <none>
I20260808 15:52:20.606011 19795 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:20.606140 19795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:20.606492 19795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:52:20.609746 19795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "47f7304d0fea46a69e5de64fe9e80943"
format_stamp: "Formatted at 2026-08-08 15:52:20 on dist-test-slave-twwt"
server_key: "a8037df5862334a09d106555b3e75cf8"
server_key_iv: "76a94c09f8d0c11d5cdad738f676cb3c"
server_key_version: "encryptionkey@0"
I20260808 15:52:20.610586 19795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "47f7304d0fea46a69e5de64fe9e80943"
format_stamp: "Formatted at 2026-08-08 15:52:20 on dist-test-slave-twwt"
server_key: "a8037df5862334a09d106555b3e75cf8"
server_key_iv: "76a94c09f8d0c11d5cdad738f676cb3c"
server_key_version: "encryptionkey@0"
I20260808 15:52:20.616056 19795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:52:20.619679 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:20.621061 19795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:52:20.621256 19795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "47f7304d0fea46a69e5de64fe9e80943"
format_stamp: "Formatted at 2026-08-08 15:52:20 on dist-test-slave-twwt"
server_key: "a8037df5862334a09d106555b3e75cf8"
server_key_iv: "76a94c09f8d0c11d5cdad738f676cb3c"
server_key_version: "encryptionkey@0"
I20260808 15:52:20.621481 19795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:20.641429 19795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:20.646235 19795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:20.646539 19795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:20.666481 19795 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:41247
I20260808 15:52:20.666460 19901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:41247 every 8 connection(s)
I20260808 15:52:20.667972 19795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:52:20.669906 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 19795
I20260808 15:52:20.670143 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:52:20.670953 27987 external_mini_cluster.cc:1428] Setting key 822957dfac091e8ab73a4f7f99cd76d2
I20260808 15:52:20.672350 19902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:52:20 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:20.687857 19902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap starting.
I20260808 15:52:20.691679 19902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:20.692802 19902 log.cc:824] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:20.695817 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:20.673073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.27.84.254:41247)
0808 15:52:20.673889 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:20.673901 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:20.673950 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:52:20.675614 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:20.675636 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:20.676053 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:20.676407 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:20.678366 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:20.679798 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:20.681815 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:20.682543 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:20.682776 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:20.687912 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:20.687922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:20.687942 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:20.687989 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:20.691287 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:20.692198 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:20.692206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:20.692213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:20.692384 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:20.692830 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:20.692836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:20.692843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:20.693468 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:20.693810 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:20.694298 (+ 488us) server_negotiation.cc:299] Negotiation successful
0808 15:52:20.694800 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":368,"threads_started":1}
I20260808 15:52:20.696105 19902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: No bootstrap required, opened a new log
I20260808 15:52:20.704264 19902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:20.704689 19902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:20.704798 19902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f7304d0fea46a69e5de64fe9e80943, State: Initialized, Role: FOLLOWER
I20260808 15:52:20.705413 19902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:20.705689 19902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:52:20.705895 19902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:52:20.706110 19902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:20.708499 19902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:20.709024 19902 leader_election.cc:302] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f7304d0fea46a69e5de64fe9e80943; no voters:
I20260808 15:52:20.709651 19902 leader_election.cc:288] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:52:20.710006 19910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:52:20.710753 19910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 LEADER]: Becoming Leader. State: Replica: 47f7304d0fea46a69e5de64fe9e80943, State: Running, Role: LEADER
I20260808 15:52:20.711279 19910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:20.711891 19902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:52:20.713979 19913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f7304d0fea46a69e5de64fe9e80943. Latest consensus state: current_term: 1 leader_uuid: "47f7304d0fea46a69e5de64fe9e80943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } } }
I20260808 15:52:20.714408 19913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: This master's current role is: LEADER
I20260808 15:52:20.714408 19911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47f7304d0fea46a69e5de64fe9e80943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } } }
I20260808 15:52:20.714826 19911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: This master's current role is: LEADER
I20260808 15:52:20.719877 19920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:52:20.725971 19920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:52:20.735728 19920 catalog_manager.cc:1347] Generated new cluster ID: 7b267f7c22524a1586d979a359603a56
I20260808 15:52:20.735924 19920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:52:20.753325 19920 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:52:20.755398 19920 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:52:20.765120 19920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Generated new TSK 0
I20260808 15:52:20.766327 19920 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:52:20.846341 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41247
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:21.207617 19955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:21.207965 19955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:21.208139 19955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:21.215926 19955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:21.216101 19955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:21.216282 19955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:52:21.227286 19955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:21.228693 19955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:21.230115 19955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:21.244028 19965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:21.244081 19966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:21.244098 19968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:21.245496 19955 server_base.cc:1034] running on GCE node
I20260808 15:52:21.246201 19955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:21.247352 19955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:21.248617 19955 hybrid_clock.cc:648] HybridClock initialized: now 1786204341248553 us; error 43 us; skew 500 ppm
I20260808 15:52:21.249068 19955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:21 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204341, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:21.252907 19955 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:52:21.254492 19955 webserver.cc:460] Webserver started at http://127.27.84.193:39071/ using document root <none> and password file <none>
I20260808 15:52:21.255338 19955 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:21.255512 19955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:21.255950 19955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:52:21.259346 19955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "2aa94237026f46e9b482864485b310b5"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "ecdf03102f35112172df703d005e190a"
server_key_iv: "d8eaa810d666c9bcb2f220311ae0f554"
server_key_version: "encryptionkey@0"
I20260808 15:52:21.260242 19955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "2aa94237026f46e9b482864485b310b5"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "ecdf03102f35112172df703d005e190a"
server_key_iv: "d8eaa810d666c9bcb2f220311ae0f554"
server_key_version: "encryptionkey@0"
I20260808 15:52:21.266100 19955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:52:21.269752 19978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:21.271186 19955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:52:21.271365 19955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "2aa94237026f46e9b482864485b310b5"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "ecdf03102f35112172df703d005e190a"
server_key_iv: "d8eaa810d666c9bcb2f220311ae0f554"
server_key_version: "encryptionkey@0"
I20260808 15:52:21.271574 19955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:21.286127 19955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:21.291217 19955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:21.291554 19955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:21.292572 19955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:21.293980 19955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:21.294086 19955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:21.294252 19955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:21.294334 19955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:21.329916 19955 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:40679
I20260808 15:52:21.329994 20138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:40679 every 8 connection(s)
I20260808 15:52:21.331506 19955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:52:21.340636 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 19955
I20260808 15:52:21.340886 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:52:21.341583 27987 external_mini_cluster.cc:1428] Setting key c6f5293a051f3b0b58f55a172a743320
Aug 08 15:52:21 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204341, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:21.354265 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:21.334546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:53511 (local address 127.27.84.254:41247)
0808 15:52:21.335113 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:21.335118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:21.336208 (+ 1090us) server_negotiation.cc:407] Connection header received
0808 15:52:21.338155 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:21.338160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:21.338261 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:21.338389 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:21.340063 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.340869 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.342562 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.343165 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.343321 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:21.347594 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:21.347600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:21.347604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:21.347641 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:21.350787 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:21.351627 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:21.351636 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:21.351640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:21.351792 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:21.352316 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:21.352321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:21.352323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:21.352607 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:21.352804 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:21.353590 (+ 786us) server_negotiation.cc:299] Negotiation successful
0808 15:52:21.353853 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":443,"thread_start_us":293,"threads_started":1}
I20260808 15:52:21.355470 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:21.335204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:53511)
0808 15:52:21.336010 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:21.336059 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:21.337854 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:21.338711 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:21.338720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:21.339284 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:52:21.339881 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:21.339899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.341057 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.341061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:21.342395 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:21.342408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.343415 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.343425 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:21.343696 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:21.344429 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:52:21.344461 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:52:21.347311 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:52:21.350968 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:21.350978 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:21.350997 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:21.351478 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:21.351925 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:21.351931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:21.351935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:21.352162 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:21.352769 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:52:21.352778 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:52:21.353159 (+ 381us) client_negotiation.cc:769] Sending connection context
0808 15:52:21.353462 (+ 303us) client_negotiation.cc:240] Negotiation successful
0808 15:52:21.354057 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":312,"threads_started":1}
I20260808 15:52:21.357398 20139 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41247
I20260808 15:52:21.357767 20139 heartbeater.cc:464] Registering TS with master...
I20260808 15:52:21.358747 20139 heartbeater.cc:511] Master 127.27.84.254:41247 requested a full tablet report, sending...
I20260808 15:52:21.361231 19852 ts_manager.cc:194] Registered new tserver with Master: 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679)
I20260808 15:52:21.362922 19852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:53511
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:52:21.413491 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41247
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:21.774178 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:21.774482 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:21.774581 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:21.782239 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:21.782368 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:21.782485 20147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:52:21.792852 20147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:21.794173 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:21.795650 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:21.809610 20161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:21.809659 20158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:21.810114 20155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:21.810890 20147 server_base.cc:1034] running on GCE node
I20260808 15:52:21.811676 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:21.812885 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:21.814119 20147 hybrid_clock.cc:648] HybridClock initialized: now 1786204341814059 us; error 56 us; skew 500 ppm
I20260808 15:52:21.814483 20147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:21 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204341, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:21.818058 20147 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:52:21.819656 20147 webserver.cc:460] Webserver started at http://127.27.84.194:36305/ using document root <none> and password file <none>
I20260808 15:52:21.820439 20147 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:21.820600 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:21.820932 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:52:21.824051 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "2fb3e9e0e980438d8eac29ab45bdd79b"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "d26b3dedb35d016b37548625b83aa343"
server_key_iv: "a5f11bcb429af6c12af2ea74e61f24db"
server_key_version: "encryptionkey@0"
I20260808 15:52:21.824872 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "2fb3e9e0e980438d8eac29ab45bdd79b"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "d26b3dedb35d016b37548625b83aa343"
server_key_iv: "a5f11bcb429af6c12af2ea74e61f24db"
server_key_version: "encryptionkey@0"
I20260808 15:52:21.830508 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:52:21.834398 20171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:21.835912 20147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:52:21.836126 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "2fb3e9e0e980438d8eac29ab45bdd79b"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "d26b3dedb35d016b37548625b83aa343"
server_key_iv: "a5f11bcb429af6c12af2ea74e61f24db"
server_key_version: "encryptionkey@0"
I20260808 15:52:21.836364 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:21.862557 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:21.867363 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:21.867681 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:21.868803 20147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:21.870231 20147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:21.870345 20147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:21.870497 20147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:21.870570 20147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:21.907236 20147 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:37763
I20260808 15:52:21.907344 20338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:37763 every 8 connection(s)
I20260808 15:52:21.908614 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:52:21.917721 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 20147
I20260808 15:52:21.917963 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:52:21.918454 27987 external_mini_cluster.cc:1428] Setting key f84117c799772b411d7eac0f92108969
Aug 08 15:52:21 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204341, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:21.931278 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:21.911381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:39949 (local address 127.27.84.254:41247)
0808 15:52:21.911916 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:21.911921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:21.912920 (+ 999us) server_negotiation.cc:407] Connection header received
0808 15:52:21.914996 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:21.915002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:21.915098 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:21.915222 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:21.916854 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.917615 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.919585 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.920145 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.920288 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:21.924569 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:21.924574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:21.924578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:21.924614 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:21.927633 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:21.928676 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:21.928680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:21.928683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:21.928855 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:21.929386 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:21.929390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:21.929392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:21.929660 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:21.929770 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:21.930780 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0808 15:52:21.930984 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":290,"threads_started":1}
I20260808 15:52:21.932884 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:21.911926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:39949)
0808 15:52:21.912728 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:21.912776 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:21.914704 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:21.915487 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:21.915496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:21.916053 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:52:21.916658 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:21.916676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.917787 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.917792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:21.919425 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:21.919438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:21.920398 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:21.920410 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:21.920666 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:21.921413 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:52:21.921437 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:52:21.924282 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:52:21.927794 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:21.927807 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:21.927828 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:21.928534 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:21.928979 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:21.928987 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:21.928993 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:21.929265 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:21.929796 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:52:21.929807 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:52:21.930301 (+ 494us) client_negotiation.cc:769] Sending connection context
0808 15:52:21.930653 (+ 352us) client_negotiation.cc:240] Negotiation successful
0808 15:52:21.931285 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":330,"threads_started":1}
I20260808 15:52:21.934538 20339 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41247
I20260808 15:52:21.934927 20339 heartbeater.cc:464] Registering TS with master...
I20260808 15:52:21.936041 20339 heartbeater.cc:511] Master 127.27.84.254:41247 requested a full tablet report, sending...
I20260808 15:52:21.937759 19852 ts_manager.cc:194] Registered new tserver with Master: 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
I20260808 15:52:21.938710 19852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:39949
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:52:21.993799 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41247
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:22.346432 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:22.346828 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:22.347015 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:22.354436 20347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:22.354602 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:22.354821 20347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:52:22.364909 20347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:22.366351 20347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:22.367007 20139 heartbeater.cc:503] Master 127.27.84.254:41247 was elected leader, sending a full tablet report...
I20260808 15:52:22.368108 20347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:22.382891 20353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:22.382823 20356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:22.382915 20354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:22.383978 20347 server_base.cc:1034] running on GCE node
I20260808 15:52:22.384631 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:22.385592 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:22.386824 20347 hybrid_clock.cc:648] HybridClock initialized: now 1786204342386731 us; error 56 us; skew 500 ppm
I20260808 15:52:22.387135 20347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:22 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204342, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:22.390456 20347 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:52:22.391973 20347 webserver.cc:460] Webserver started at http://127.27.84.195:33939/ using document root <none> and password file <none>
I20260808 15:52:22.392720 20347 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:22.392864 20347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:22.393193 20347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:52:22.396309 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "633727d132ce4612b5fabbdca0ef1056"
format_stamp: "Formatted at 2026-08-08 15:52:22 on dist-test-slave-twwt"
server_key: "41daf653703b70e94c371688abff05c2"
server_key_iv: "d0df808767472e442ffd454e48f536fc"
server_key_version: "encryptionkey@0"
I20260808 15:52:22.397082 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "633727d132ce4612b5fabbdca0ef1056"
format_stamp: "Formatted at 2026-08-08 15:52:22 on dist-test-slave-twwt"
server_key: "41daf653703b70e94c371688abff05c2"
server_key_iv: "d0df808767472e442ffd454e48f536fc"
server_key_version: "encryptionkey@0"
I20260808 15:52:22.402385 20347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:52:22.406116 20368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:22.407649 20347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:52:22.407828 20347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "633727d132ce4612b5fabbdca0ef1056"
format_stamp: "Formatted at 2026-08-08 15:52:22 on dist-test-slave-twwt"
server_key: "41daf653703b70e94c371688abff05c2"
server_key_iv: "d0df808767472e442ffd454e48f536fc"
server_key_version: "encryptionkey@0"
I20260808 15:52:22.408030 20347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:22.461762 20347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:22.466720 20347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:22.467049 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:22.468127 20347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:22.469643 20347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:22.469725 20347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:22.469894 20347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:22.469956 20347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:22.504926 20347 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:45697
I20260808 15:52:22.505024 20529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:45697 every 8 connection(s)
I20260808 15:52:22.506315 20347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:52:22.516657 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 20347
I20260808 15:52:22.516870 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:52:22.517328 27987 external_mini_cluster.cc:1428] Setting key 6bf0dc795a115ac3661d3ca281d52fe8
Aug 08 15:52:22 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204342, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:22.527985 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.509114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51423 (local address 127.27.84.254:41247)
0808 15:52:22.509598 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.509603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.510663 (+ 1060us) server_negotiation.cc:407] Connection header received
0808 15:52:22.512807 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.512812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.512908 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.513032 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:22.514556 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.515402 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.516990 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.517408 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.517560 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.521806 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:22.521812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:22.521817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:22.521853 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:22.524818 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:22.525582 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:22.525586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:22.525588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:22.525751 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:22.526210 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:22.526215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:22.526217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:22.526575 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:22.526787 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.527486 (+ 699us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.527661 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":239,"threads_started":1}
I20260808 15:52:22.528991 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.509701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:51423)
0808 15:52:22.510454 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:22.510502 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:22.512476 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:22.513280 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:22.513288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:22.513820 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:52:22.514376 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.514398 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.515564 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.515570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.516838 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.516856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.517657 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.517664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.517907 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.518650 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:52:22.518684 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:52:22.521610 (+ 2926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:52:22.524951 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:22.524957 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:22.524969 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:22.525458 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:22.525851 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:22.525854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:22.525857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:22.526058 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:22.526781 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:52:22.526788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:52:22.527124 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 15:52:22.527366 (+ 242us) client_negotiation.cc:240] Negotiation successful
0808 15:52:22.527776 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":294,"threads_started":1}
I20260808 15:52:22.530519 20531 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41247
I20260808 15:52:22.530892 20531 heartbeater.cc:464] Registering TS with master...
I20260808 15:52:22.531785 20531 heartbeater.cc:511] Master 127.27.84.254:41247 requested a full tablet report, sending...
I20260808 15:52:22.533628 19852 ts_manager.cc:194] Registered new tserver with Master: 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697)
I20260808 15:52:22.534461 19852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:51423
I20260808 15:52:22.545897 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:52:22.564947 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.550964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.27.84.254:41247)
0808 15:52:22.551193 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.551218 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.551723 (+ 505us) server_negotiation.cc:407] Connection header received
0808 15:52:22.552003 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.552008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.552101 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.552224 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:22.554344 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.555317 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.556716 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.557116 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.557217 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.559767 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:22.559771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:22.559775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:22.559806 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:22.562355 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:22.563140 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:22.563145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:22.563147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:22.563284 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:22.563677 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:22.563681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:22.563683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:22.564024 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:22.564216 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.564515 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.564689 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:52:22.569265 19852 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:52:22.576623 19852 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:52:22.613660 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.593410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.27.84.193:40679)
0808 15:52:22.604946 (+ 11536us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.604957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.605564 (+ 607us) server_negotiation.cc:407] Connection header received
0808 15:52:22.605748 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.605755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.605986 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.606169 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:22.607070 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.608584 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.610431 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.612076 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.612224 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.612486 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.613114 (+ 628us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.613293 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11322,"thread_start_us":11230,"threads_started":1}
I20260808 15:52:22.613725 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.589993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.27.84.195:45697)
0808 15:52:22.602672 (+ 12679us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.602686 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.603722 (+ 1036us) server_negotiation.cc:407] Connection header received
0808 15:52:22.604421 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.604431 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.604763 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.605775 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:22.606095 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.607717 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.610587 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.612076 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.612546 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.612738 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.613241 (+ 503us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.613398 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12384,"thread_start_us":12171,"threads_started":1}
I20260808 15:52:22.613817 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.592663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:39284)
0808 15:52:22.605431 (+ 12768us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:22.605450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:22.605583 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:22.606319 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:22.606324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:22.606352 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:22.606853 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.606871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.608726 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.608731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.610264 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.610283 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.612321 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.612329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.612553 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.612824 (+ 271us) client_negotiation.cc:769] Sending connection context
0808 15:52:22.613054 (+ 230us) client_negotiation.cc:240] Negotiation successful
0808 15:52:22.613319 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12642,"mutex_wait_us":31,"thread_start_us":12210,"threads_started":1}
I20260808 15:52:22.614229 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.590064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37763 (local address 127.0.0.1:45498)
0808 15:52:22.605039 (+ 14975us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:22.605060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:22.605243 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:22.605985 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:22.605989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:22.606011 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:22.606426 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.606439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.608466 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.608471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.611166 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.611181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.613392 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.613398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.613585 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.613627 (+ 42us) client_negotiation.cc:769] Sending connection context
0808 15:52:22.613725 (+ 98us) client_negotiation.cc:240] Negotiation successful
0808 15:52:22.613888 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14835,"mutex_wait_us":53,"spinlock_wait_cycles":896,"thread_start_us":15067,"threads_started":1}
I20260808 15:52:22.614743 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.589615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45697 (local address 127.0.0.1:45096)
0808 15:52:22.603196 (+ 13581us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:22.603239 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:22.603496 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:22.605086 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:22.605100 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:22.605151 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:22.605666 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.605680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.607889 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.607895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.610430 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.610446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.612375 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.612381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.612914 (+ 533us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.612965 (+ 51us) client_negotiation.cc:769] Sending connection context
0808 15:52:22.613080 (+ 115us) client_negotiation.cc:240] Negotiation successful
0808 15:52:22.614303 (+ 1223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13310,"mutex_wait_us":1130,"spinlock_wait_cycles":27264,"thread_start_us":13130,"threads_started":1}
I20260808 15:52:22.615091 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.590089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.27.84.194:37763)
0808 15:52:22.604247 (+ 14158us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.604256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.605397 (+ 1141us) server_negotiation.cc:407] Connection header received
0808 15:52:22.605481 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.605486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.605765 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.605977 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:22.606665 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.608309 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.611402 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.612900 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.613052 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.613248 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.614555 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.614780 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13981,"thread_start_us":13872,"threads_started":1}
I20260808 15:52:22.618007 20041 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1761b743c24a3a81cec0dc4ffa2c55 (DEFAULT_TABLE table=test-table [id=05332ba69a6743f8a99cc79247e57d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:52:22.618010 20238 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1761b743c24a3a81cec0dc4ffa2c55 (DEFAULT_TABLE table=test-table [id=05332ba69a6743f8a99cc79247e57d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:52:22.619426 20238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1761b743c24a3a81cec0dc4ffa2c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:52:22.619506 20041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1761b743c24a3a81cec0dc4ffa2c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:52:22.622668 20435 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1761b743c24a3a81cec0dc4ffa2c55 (DEFAULT_TABLE table=test-table [id=05332ba69a6743f8a99cc79247e57d25]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:52:22.624398 20435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1761b743c24a3a81cec0dc4ffa2c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:52:22.631807 20563 tablet_bootstrap.cc:492] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Bootstrap starting.
I20260808 15:52:22.635762 20563 tablet_bootstrap.cc:654] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:22.635936 20564 tablet_bootstrap.cc:492] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Bootstrap starting.
I20260808 15:52:22.637302 20563 log.cc:824] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:22.639693 20564 tablet_bootstrap.cc:654] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:22.639977 20568 tablet_bootstrap.cc:492] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Bootstrap starting.
I20260808 15:52:22.640883 20563 tablet_bootstrap.cc:492] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: No bootstrap required, opened a new log
I20260808 15:52:22.641175 20563 ts_tablet_manager.cc:1397] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 15:52:22.641177 20564 log.cc:824] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:22.643501 20568 tablet_bootstrap.cc:654] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:22.644574 20564 tablet_bootstrap.cc:492] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: No bootstrap required, opened a new log
I20260808 15:52:22.644899 20564 ts_tablet_manager.cc:1397] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:52:22.644899 20568 log.cc:824] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:22.648002 20568 tablet_bootstrap.cc:492] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: No bootstrap required, opened a new log
I20260808 15:52:22.648260 20568 ts_tablet_manager.cc:1397] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:52:22.649667 20563 raft_consensus.cc:348] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.650032 20563 raft_consensus.cc:374] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:22.650161 20563 raft_consensus.cc:729] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa94237026f46e9b482864485b310b5, State: Initialized, Role: FOLLOWER
I20260808 15:52:22.650811 20563 consensus_queue.cc:260] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.653772 20563 ts_tablet_manager.cc:1428] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260808 15:52:22.655754 20564 raft_consensus.cc:348] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.656204 20564 raft_consensus.cc:374] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:22.656327 20564 raft_consensus.cc:729] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb3e9e0e980438d8eac29ab45bdd79b, State: Initialized, Role: FOLLOWER
I20260808 15:52:22.656935 20564 consensus_queue.cc:260] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.658142 20339 heartbeater.cc:503] Master 127.27.84.254:41247 was elected leader, sending a full tablet report...
I20260808 15:52:22.658875 20564 ts_tablet_manager.cc:1428] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260808 15:52:22.659132 20568 raft_consensus.cc:348] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.659631 20568 raft_consensus.cc:374] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:22.659732 20568 raft_consensus.cc:729] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633727d132ce4612b5fabbdca0ef1056, State: Initialized, Role: FOLLOWER
I20260808 15:52:22.660331 20568 consensus_queue.cc:260] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
W20260808 15:52:22.660772 20340 tablet.cc:2369] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:52:22.661481 20531 heartbeater.cc:503] Master 127.27.84.254:41247 was elected leader, sending a full tablet report...
I20260808 15:52:22.662187 20568 ts_tablet_manager.cc:1428] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
W20260808 15:52:22.758152 20533 tablet.cc:2369] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:52:22.834347 20140 tablet.cc:2369] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:52:22.874811 20574 raft_consensus.cc:482] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:52:22.875147 20574 raft_consensus.cc:504] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.877115 20574 leader_election.cc:288] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697), 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679)
I20260808 15:52:22.884446 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.877600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:45355 (local address 127.27.84.195:45697)
0808 15:52:22.877849 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.877859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.878871 (+ 1012us) server_negotiation.cc:407] Connection header received
0808 15:52:22.879160 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.879165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.879227 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.879327 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:22.880029 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.880766 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.882702 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.883484 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.883682 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.883759 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.884035 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.884167 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:52:22.884634 20577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.877614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45697 (local address 127.27.84.194:45355)
0808 15:52:22.878743 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:22.878758 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:22.879000 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:22.879346 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:22.879350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:22.879374 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:22.879857 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.879867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.880913 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.880929 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.882521 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.882533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.883709 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.883715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.883864 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.883888 (+ 24us) client_negotiation.cc:769] Sending connection context
0808 15:52:22.884004 (+ 116us) client_negotiation.cc:240] Negotiation successful
0808 15:52:22.884152 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"mutex_wait_us":102,"spinlock_wait_cycles":80768,"thread_start_us":572,"threads_started":1}
I20260808 15:52:22.884931 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.877654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:58785 (local address 127.27.84.193:40679)
0808 15:52:22.877864 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.877872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.878847 (+ 975us) server_negotiation.cc:407] Connection header received
0808 15:52:22.879098 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.879102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.879171 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.879281 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:22.880089 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.880900 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.882970 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.883863 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.884060 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.884145 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.884507 (+ 362us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.884648 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:52:22.885099 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.877592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.27.84.194:58785)
0808 15:52:22.878697 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:22.878721 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:22.878910 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:22.879312 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:22.879317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:22.879346 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:22.879916 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.879926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.881054 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.881061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.882783 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:22.882798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.884108 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.884115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:22.884318 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.884343 (+ 25us) client_negotiation.cc:769] Sending connection context
0808 15:52:22.884475 (+ 132us) client_negotiation.cc:240] Negotiation successful
0808 15:52:22.884630 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":3,"thread_start_us":482,"threads_started":1}
I20260808 15:52:22.886245 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1761b743c24a3a81cec0dc4ffa2c55" candidate_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633727d132ce4612b5fabbdca0ef1056" is_pre_election: true
I20260808 15:52:22.886252 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1761b743c24a3a81cec0dc4ffa2c55" candidate_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa94237026f46e9b482864485b310b5" is_pre_election: true
I20260808 15:52:22.886983 20074 raft_consensus.cc:2454] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb3e9e0e980438d8eac29ab45bdd79b in term 0.
I20260808 15:52:22.886978 20463 raft_consensus.cc:2454] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb3e9e0e980438d8eac29ab45bdd79b in term 0.
I20260808 15:52:22.887912 20174 leader_election.cc:302] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2aa94237026f46e9b482864485b310b5, 2fb3e9e0e980438d8eac29ab45bdd79b, 633727d132ce4612b5fabbdca0ef1056; no voters:
I20260808 15:52:22.888471 20574 raft_consensus.cc:2781] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:52:22.888607 20574 raft_consensus.cc:482] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:52:22.888747 20574 raft_consensus.cc:3037] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:22.891109 20574 raft_consensus.cc:504] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.891749 20574 leader_election.cc:288] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [CANDIDATE]: Term 1 election: Requested vote from peers 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697), 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679)
I20260808 15:52:22.892375 20463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1761b743c24a3a81cec0dc4ffa2c55" candidate_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633727d132ce4612b5fabbdca0ef1056"
I20260808 15:52:22.892369 20074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d1761b743c24a3a81cec0dc4ffa2c55" candidate_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa94237026f46e9b482864485b310b5"
I20260808 15:52:22.892612 20074 raft_consensus.cc:3037] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:22.892612 20463 raft_consensus.cc:3037] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:22.896081 20463 raft_consensus.cc:2454] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb3e9e0e980438d8eac29ab45bdd79b in term 1.
I20260808 15:52:22.896081 20074 raft_consensus.cc:2454] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb3e9e0e980438d8eac29ab45bdd79b in term 1.
I20260808 15:52:22.896639 20173 leader_election.cc:302] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2aa94237026f46e9b482864485b310b5, 2fb3e9e0e980438d8eac29ab45bdd79b, 633727d132ce4612b5fabbdca0ef1056; no voters:
I20260808 15:52:22.897042 20574 raft_consensus.cc:2781] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:52:22.897547 20574 raft_consensus.cc:686] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 LEADER]: Becoming Leader. State: Replica: 2fb3e9e0e980438d8eac29ab45bdd79b, State: Running, Role: LEADER
I20260808 15:52:22.898051 20574 consensus_queue.cc:237] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } }
I20260808 15:52:22.903256 19852 catalog_manager.cc:5495] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194). New cstate: current_term: 1 leader_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } health_report { overall_health: UNKNOWN } } }
I20260808 15:52:22.984377 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:22.978479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.27.84.194:37763)
0808 15:52:22.978735 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:22.978742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:22.978814 (+ 72us) server_negotiation.cc:407] Connection header received
0808 15:52:22.979095 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:22.979099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:22.979196 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:22.979361 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:52:22.979780 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.980518 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.982028 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:22.982457 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:22.982600 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:22.983194 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:52:22.983713 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:52:22.983905 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:22.983989 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 15:52:22.984080 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:52:22.992856 20463 raft_consensus.cc:1261] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Refusing update from remote peer 2fb3e9e0e980438d8eac29ab45bdd79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:52:22.992885 20074 raft_consensus.cc:1261] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Refusing update from remote peer 2fb3e9e0e980438d8eac29ab45bdd79b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:52:22.993868 20578 consensus_queue.cc:1035] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:52:22.994223 20574 consensus_queue.cc:1035] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:52:23.024497 19852 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:52:23.024812 19852 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:52:23.028878 19852 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Sending DeleteTablet for 3 replicas of tablet 8d1761b743c24a3a81cec0dc4ffa2c55
I20260808 15:52:23.030239 20041 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1761b743c24a3a81cec0dc4ffa2c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:23 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:39284
I20260808 15:52:23.030241 20435 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1761b743c24a3a81cec0dc4ffa2c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:23 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45096
I20260808 15:52:23.031219 20598 tablet_replica.cc:330] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: stopping tablet replica
I20260808 15:52:23.031431 20238 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1761b743c24a3a81cec0dc4ffa2c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:23 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:45498
I20260808 15:52:23.031883 20599 tablet_replica.cc:330] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: stopping tablet replica
I20260808 15:52:23.032392 20599 raft_consensus.cc:2229] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:52:23.032663 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 19955
I20260808 15:52:23.032986 20599 raft_consensus.cc:2258] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:23.033710 20598 raft_consensus.cc:2229] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:52:23.034190 20598 raft_consensus.cc:2258] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:23.036499 20598 ts_tablet_manager.cc:1905] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:23.043448 20600 tablet_replica.cc:330] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: stopping tablet replica
I20260808 15:52:23.044068 20600 raft_consensus.cc:2229] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:52:23.044679 20598 ts_tablet_manager.cc:1918] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:23.044860 20598 log.cc:1196] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/8d1761b743c24a3a81cec0dc4ffa2c55
I20260808 15:52:23.044878 20600 raft_consensus.cc:2258] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:23.045445 20598 ts_tablet_manager.cc:1939] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 633727d132ce4612b5fabbdca0ef1056: Deleting consensus metadata
I20260808 15:52:23.046056 20599 ts_tablet_manager.cc:1905] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:23.047328 20600 ts_tablet_manager.cc:1905] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:23.048751 19828 catalog_manager.cc:4841] TS 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697): tablet 8d1761b743c24a3a81cec0dc4ffa2c55 (table test-table [id=05332ba69a6743f8a99cc79247e57d25]) successfully deleted
I20260808 15:52:23.055183 20600 ts_tablet_manager.cc:1918] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:23.055402 20600 log.cc:1196] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/8d1761b743c24a3a81cec0dc4ffa2c55
I20260808 15:52:23.055591 20599 ts_tablet_manager.cc:1918] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:23.055810 20599 log.cc:1196] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/8d1761b743c24a3a81cec0dc4ffa2c55
I20260808 15:52:23.056008 20600 ts_tablet_manager.cc:1939] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2fb3e9e0e980438d8eac29ab45bdd79b: Deleting consensus metadata
I20260808 15:52:23.056483 20599 ts_tablet_manager.cc:1939] T 8d1761b743c24a3a81cec0dc4ffa2c55 P 2aa94237026f46e9b482864485b310b5: Deleting consensus metadata
I20260808 15:52:23.058122 19830 catalog_manager.cc:4841] TS 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763): tablet 8d1761b743c24a3a81cec0dc4ffa2c55 (table test-table [id=05332ba69a6743f8a99cc79247e57d25]) successfully deleted
I20260808 15:52:23.058370 19827 catalog_manager.cc:4841] TS 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679): tablet 8d1761b743c24a3a81cec0dc4ffa2c55 (table test-table [id=05332ba69a6743f8a99cc79247e57d25]) successfully deleted
Aug 08 15:52:23 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:52:23.073442 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:23.034407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.27.84.193:40679)
0808 15:52:23.034656 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:23.034665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:23.048413 (+ 13748us) server_negotiation.cc:407] Connection header received
0808 15:52:23.054839 (+ 6426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:23.054846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:23.054958 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:23.055042 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:23.057372 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:23.058573 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:23.060745 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:23.061289 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:23.061399 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:23.067097 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:23.067110 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:23.067126 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:23.067168 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:23.070164 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:23.071111 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:23.071120 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:23.071122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:23.071276 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:23.071704 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:23.071707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:23.071710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:23.072111 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:23.072443 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:23.072834 (+ 391us) server_negotiation.cc:299] Negotiation successful
0808 15:52:23.073072 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:52:23.074167 20133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:23.166673 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 19955
I20260808 15:52:23.197979 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 20147
Aug 08 15:52:23 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:52:23.216179 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:23.198870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.27.84.194:37763)
0808 15:52:23.199090 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:23.199099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:23.199161 (+ 62us) server_negotiation.cc:407] Connection header received
0808 15:52:23.199420 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:23.199424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:23.199510 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:23.199619 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:23.201009 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:23.201804 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:23.203477 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:23.203952 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:23.204105 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:23.208836 (+ 4731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:23.208852 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:23.208868 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:23.208907 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:23.212963 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:23.213817 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:23.213827 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:23.213830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:23.214001 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:23.214410 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:23.214414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:23.214418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:23.214916 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:23.215264 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:23.215507 (+ 243us) server_negotiation.cc:299] Negotiation successful
0808 15:52:23.215754 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:52:23.216951 20332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:23.334766 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 20147
I20260808 15:52:23.362370 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 20347
Aug 08 15:52:23 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:52:23.377801 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:23.363212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.27.84.195:45697)
0808 15:52:23.363412 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:23.363417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:23.363438 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:52:23.363585 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:23.363589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:23.363653 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:23.363738 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:23.365060 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:23.365771 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:23.367161 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:23.367548 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:23.367643 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:23.372303 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:23.372317 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:23.372334 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:23.372373 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:23.375117 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:23.375752 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:23.375759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:23.375761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:23.375894 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:23.376259 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:23.376262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:23.376264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:23.376648 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:23.376877 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:23.377243 (+ 366us) server_negotiation.cc:299] Negotiation successful
0808 15:52:23.377410 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:52:23.378561 20525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:23.469173 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 20347
I20260808 15:52:23.495981 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 19795
I20260808 15:52:23.497166 19893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:23.612476 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 19795
I20260808 15:52:43.644345 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41247
--webserver_interface=127.27.84.254
--webserver_port=40731
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:52:44.017679 20609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:44.018029 20609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:44.018208 20609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:44.025755 20609 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:52:44.025882 20609 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:52:44.025957 20609 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:52:44.026044 20609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:52:44.026119 20609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:44.026237 20609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:44.026314 20609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:52:44.026400 20609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:52:44.037233 20609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:41247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=40731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:44.038758 20609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:44.040249 20609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:44.068905 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:44.068825 20615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:44.068958 20616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:44.069352 20609 server_base.cc:1034] running on GCE node
I20260808 15:52:44.070838 20609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:44.072613 20609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:44.073925 20609 hybrid_clock.cc:648] HybridClock initialized: now 1786204364073859 us; error 63 us; skew 500 ppm
I20260808 15:52:44.074366 20609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:44 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:44.083269 20609 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:52:44.085271 20609 webserver.cc:460] Webserver started at http://127.27.84.254:40731/ using document root <none> and password file <none>
I20260808 15:52:44.086170 20609 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:44.086341 20609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:44.093142 20609 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260808 15:52:44.096916 20630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:44.098382 20609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:52:44.098690 20609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "47f7304d0fea46a69e5de64fe9e80943"
format_stamp: "Formatted at 2026-08-08 15:52:20 on dist-test-slave-twwt"
server_key: "a8037df5862334a09d106555b3e75cf8"
server_key_iv: "76a94c09f8d0c11d5cdad738f676cb3c"
server_key_version: "encryptionkey@0"
I20260808 15:52:44.099897 20609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:44.113905 20609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:44.118491 20609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:44.118865 20609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:44.140494 20609 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:41247
I20260808 15:52:44.140519 20720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:41247 every 8 connection(s)
I20260808 15:52:44.142292 20609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:52:44.150139 20721 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:52:44.151409 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 20609
I20260808 15:52:44.152287 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:40679
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:41247
--webserver_port=39071
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:52:44.153934 20721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap starting.
I20260808 15:52:44.176270 20721 log.cc:824] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:44.189280 20721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:52:44.189880 20721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap complete.
I20260808 15:52:44.199841 20721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:44.200778 20721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f7304d0fea46a69e5de64fe9e80943, State: Initialized, Role: FOLLOWER
I20260808 15:52:44.201411 20721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:44.201682 20721 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:52:44.201829 20721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:52:44.202018 20721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 1 FOLLOWER]: Advancing to term 2
I20260808 15:52:44.206379 20721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:44.206990 20721 leader_election.cc:302] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f7304d0fea46a69e5de64fe9e80943; no voters:
I20260808 15:52:44.207690 20721 leader_election.cc:288] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260808 15:52:44.207851 20727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Leader election won for term 2
I20260808 15:52:44.209077 20727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 LEADER]: Becoming Leader. State: Replica: 47f7304d0fea46a69e5de64fe9e80943, State: Running, Role: LEADER
I20260808 15:52:44.209717 20727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:52:44.210193 20721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:52:44.212579 20730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f7304d0fea46a69e5de64fe9e80943. Latest consensus state: current_term: 2 leader_uuid: "47f7304d0fea46a69e5de64fe9e80943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } } }
I20260808 15:52:44.212626 20729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47f7304d0fea46a69e5de64fe9e80943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } } }
I20260808 15:52:44.213055 20730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: This master's current role is: LEADER
I20260808 15:52:44.213106 20729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: This master's current role is: LEADER
I20260808 15:52:44.214105 20735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:52:44.230547 20735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:52:44.232137 20735 catalog_manager.cc:1259] Loaded cluster ID: 7b267f7c22524a1586d979a359603a56
I20260808 15:52:44.232254 20735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:52:44.237030 20735 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:52:44.239209 20735 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Loaded TSK: 0
I20260808 15:52:44.278218 20735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Generated new TSK 1
I20260808 15:52:44.278730 20735 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:52:44.551702 20723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:44.552045 20723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:44.552215 20723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:44.559710 20723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:44.559860 20723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:44.560032 20723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:52:44.570156 20723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:40679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=39071
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:44.571620 20723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:44.573002 20723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:44.588289 20763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:44.588765 20761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:44.589421 20723 server_base.cc:1034] running on GCE node
W20260808 15:52:44.590850 20765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:44.591609 20723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:44.592738 20723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:44.593991 20723 hybrid_clock.cc:648] HybridClock initialized: now 1786204364593945 us; error 32 us; skew 500 ppm
I20260808 15:52:44.594357 20723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:44 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:44.598120 20723 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:52:44.600029 20723 webserver.cc:460] Webserver started at http://127.27.84.193:39071/ using document root <none> and password file <none>
I20260808 15:52:44.601050 20723 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:44.601250 20723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:44.608502 20723 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260808 15:52:44.618316 20774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:44.620172 20723 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.005s
I20260808 15:52:44.620359 20723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "2aa94237026f46e9b482864485b310b5"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "ecdf03102f35112172df703d005e190a"
server_key_iv: "d8eaa810d666c9bcb2f220311ae0f554"
server_key_version: "encryptionkey@0"
I20260808 15:52:44.621769 20723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:44.666548 20723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:44.673461 20723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:44.673831 20723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:44.675323 20723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:44.677516 20723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:44.677651 20723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:44.677786 20723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:44.677891 20723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:44.715605 20723 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:40679
I20260808 15:52:44.715700 20944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:40679 every 8 connection(s)
I20260808 15:52:44.717178 20723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:52:44.718904 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 20723
I20260808 15:52:44.719761 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:37763
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:41247
--webserver_port=36305
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:52:44 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204364, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:44.746151 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:44.720927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:34117)
0808 15:52:44.721904 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:44.721959 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:44.724765 (+ 2806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:44.726289 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:44.726303 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:44.727264 (+ 961us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:52:44.728105 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:44.728134 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:44.730041 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:44.730049 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:44.732312 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:44.732331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:44.733388 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:44.733398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:44.733718 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:44.734460 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:52:44.734490 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:52:44.736970 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:52:44.741593 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:44.741601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:44.741613 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:44.742114 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:44.742714 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:44.742718 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:44.742720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:44.742932 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:44.744012 (+ 1080us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:52:44.744024 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:52:44.744360 (+ 336us) client_negotiation.cc:769] Sending connection context
0808 15:52:44.744618 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 15:52:44.745064 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":337,"threads_started":1}
I20260808 15:52:44.746734 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:44.720597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:34117 (local address 127.27.84.254:41247)
0808 15:52:44.721254 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:44.721277 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:44.722132 (+ 855us) server_negotiation.cc:407] Connection header received
0808 15:52:44.725353 (+ 3221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:44.725366 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:44.725716 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:44.726034 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:44.728379 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:44.729859 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:44.732471 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:44.733128 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:44.733274 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:44.737244 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:44.737256 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:44.737279 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:44.737338 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:44.741408 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:44.742289 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:44.742301 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:44.742312 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:44.742535 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:44.743079 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:44.743087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:44.743094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:44.743845 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:44.744052 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:44.744906 (+ 854us) server_negotiation.cc:299] Negotiation successful
0808 15:52:44.745374 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":287,"threads_started":1}
I20260808 15:52:44.749811 20945 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41247
I20260808 15:52:44.750182 20945 heartbeater.cc:464] Registering TS with master...
I20260808 15:52:44.751226 20945 heartbeater.cc:511] Master 127.27.84.254:41247 requested a full tablet report, sending...
I20260808 15:52:44.753763 20656 ts_manager.cc:194] Registered new tserver with Master: 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679)
I20260808 15:52:44.755367 20656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:34117
W20260808 15:52:45.086256 20951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:45.086596 20951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:45.086824 20951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:45.094244 20951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:45.094359 20951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:45.094472 20951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:52:45.104423 20951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:37763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=36305
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:45.105882 20951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:45.107231 20951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:45.121251 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:45.121316 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:45.121343 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:45.122262 20951 server_base.cc:1034] running on GCE node
I20260808 15:52:45.123063 20951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:45.124228 20951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:45.125458 20951 hybrid_clock.cc:648] HybridClock initialized: now 1786204365125409 us; error 36 us; skew 500 ppm
I20260808 15:52:45.125865 20951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:45 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204365, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:45.129566 20951 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:52:45.131297 20951 webserver.cc:460] Webserver started at http://127.27.84.194:36305/ using document root <none> and password file <none>
I20260808 15:52:45.132046 20951 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:45.132218 20951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:45.138222 20951 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260808 15:52:45.141408 20974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:45.143092 20951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:52:45.143265 20951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "2fb3e9e0e980438d8eac29ab45bdd79b"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "d26b3dedb35d016b37548625b83aa343"
server_key_iv: "a5f11bcb429af6c12af2ea74e61f24db"
server_key_version: "encryptionkey@0"
I20260808 15:52:45.144624 20951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:45.166153 20951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:45.171073 20951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:45.171357 20951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:45.172446 20951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:45.173933 20951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:45.174033 20951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:45.174122 20951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:45.174187 20951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:45.212036 20951 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:37763
I20260808 15:52:45.212087 21151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:37763 every 8 connection(s)
I20260808 15:52:45.213786 20951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:52:45.220062 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 20951
I20260808 15:52:45.220917 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:45697
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:41247
--webserver_port=33939
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 08 15:52:45 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204365, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:45.239886 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.216720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:46043 (local address 127.27.84.254:41247)
0808 15:52:45.216940 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.216945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.218310 (+ 1365us) server_negotiation.cc:407] Connection header received
0808 15:52:45.220861 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.220867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.220965 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.221128 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:45.223601 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.224436 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.227239 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.227806 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.227921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.232966 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:45.232975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:45.232983 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:45.233031 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:45.236426 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.237275 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.237280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.237282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.237435 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.237926 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.237936 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.237939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.238240 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:45.238419 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.239250 (+ 831us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.239530 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:52:45.240998 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.217272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:46043)
0808 15:52:45.218116 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.218163 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.220474 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.221442 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.221457 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.222360 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:52:45.223342 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.223372 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.224639 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.224647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.227001 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.227021 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.228067 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.228077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.228451 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.229314 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:52:45.229352 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:52:45.232583 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:52:45.236588 (+ 4005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:45.236596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:45.236612 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:45.237102 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:45.237565 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:45.237570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:45.237576 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:45.237787 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:45.238384 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:52:45.238394 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:52:45.238849 (+ 455us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.239145 (+ 296us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.239758 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":312,"threads_started":1}
I20260808 15:52:45.242875 21152 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41247
I20260808 15:52:45.243217 21152 heartbeater.cc:464] Registering TS with master...
I20260808 15:52:45.244036 21152 heartbeater.cc:511] Master 127.27.84.254:41247 requested a full tablet report, sending...
I20260808 15:52:45.245723 20656 ts_manager.cc:194] Registered new tserver with Master: 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
I20260808 15:52:45.247092 20656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:46043
W20260808 15:52:45.584075 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:52:45.584368 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:52:45.584512 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:52:45.592029 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:52:45.592144 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:52:45.592324 21155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:52:45.602581 21155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:45697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=33939
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:52:45.604067 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:52:45.605551 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:52:45.619865 21169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:45.619841 21175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:52:45.619797 21171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:52:45.620748 21155 server_base.cc:1034] running on GCE node
I20260808 15:52:45.621398 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:52:45.622361 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:52:45.623610 21155 hybrid_clock.cc:648] HybridClock initialized: now 1786204365623552 us; error 48 us; skew 500 ppm
I20260808 15:52:45.624018 21155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:52:45 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204365, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:52:45.627626 21155 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:52:45.629475 21155 webserver.cc:460] Webserver started at http://127.27.84.195:33939/ using document root <none> and password file <none>
I20260808 15:52:45.630295 21155 fs_manager.cc:359] Metadata directory not provided
I20260808 15:52:45.630473 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:52:45.636787 21155 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260808 15:52:45.640317 21184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:45.642203 21155 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260808 15:52:45.642354 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "633727d132ce4612b5fabbdca0ef1056"
format_stamp: "Formatted at 2026-08-08 15:52:22 on dist-test-slave-twwt"
server_key: "41daf653703b70e94c371688abff05c2"
server_key_iv: "d0df808767472e442ffd454e48f536fc"
server_key_version: "encryptionkey@0"
I20260808 15:52:45.643445 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:52:45.657115 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:52:45.661522 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:52:45.661796 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:52:45.662915 21155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:52:45.664383 21155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:52:45.664481 21155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:45.664570 21155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:52:45.664636 21155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:52:45.702292 21155 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:45697
I20260808 15:52:45.702343 21344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:45697 every 8 connection(s)
I20260808 15:52:45.703995 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:52:45.709007 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 21155
Aug 08 15:52:45 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204365, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:52:45.731588 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.707162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:51637 (local address 127.27.84.254:41247)
0808 15:52:45.707397 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.707404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.708668 (+ 1264us) server_negotiation.cc:407] Connection header received
0808 15:52:45.711803 (+ 3135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.711813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.711940 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.712046 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:45.715261 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.716585 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.718991 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.719763 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.719896 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.724085 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:45.724093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:45.724098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:45.724143 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:45.727501 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.728594 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.728600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.728604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.728779 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.729413 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.729418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.729421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.729785 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:45.729952 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.730940 (+ 988us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.731180 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:52:45.731956 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.710096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48372 (local address 127.27.84.254:41247)
0808 15:52:45.715053 (+ 4957us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.715058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.715079 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:52:45.715133 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.715138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.715228 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.715352 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:45.717503 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.718326 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.720457 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.721105 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.721281 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.724569 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:45.724574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:45.724577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:45.724620 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:45.728416 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.729463 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.729467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.729469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.729631 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.730164 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.730168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.730171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.730663 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:45.730932 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.731351 (+ 419us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.731550 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4832,"thread_start_us":4709,"threads_started":1}
I20260808 15:52:45.732800 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.707678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:51637)
0808 15:52:45.708474 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.708519 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.711333 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.712821 (+ 1488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.712839 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.713931 (+ 1092us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:52:45.715016 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.715071 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.716784 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.716790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.718825 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.718841 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.720020 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.720029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.720356 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.721258 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:52:45.721289 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:52:45.723826 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:52:45.727655 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:45.727665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:45.727685 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:45.728413 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:45.728933 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:52:45.728936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:52:45.728939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:52:45.729278 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:52:45.729925 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:52:45.729932 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:52:45.730409 (+ 477us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.730820 (+ 411us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.731385 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":303,"threads_started":1}
I20260808 15:52:45.734437 21346 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41247
I20260808 15:52:45.734829 21346 heartbeater.cc:464] Registering TS with master...
I20260808 15:52:45.735638 21346 heartbeater.cc:511] Master 127.27.84.254:41247 requested a full tablet report, sending...
I20260808 15:52:45.737203 20653 ts_manager.cc:194] Registered new tserver with Master: 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697)
I20260808 15:52:45.738157 20653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:51637
I20260808 15:52:45.748301 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:52:45.756138 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.749392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.27.84.254:41247)
0808 15:52:45.749676 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.749680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.750246 (+ 566us) server_negotiation.cc:407] Connection header received
0808 15:52:45.750524 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.750528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.750757 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.750915 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:52:45.751400 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.752160 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.753801 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.754310 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.754429 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.755150 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:52:45.755460 (+ 310us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0808 15:52:45.755845 (+ 385us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48388: token expired
Metrics: {"server-negotiator.queue_time_us":120}
W20260808 15:52:45.756268 21351 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48388: token expired
W20260808 15:52:45.756917 20542 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.84.254:41247)
I20260808 15:52:45.759728 20945 heartbeater.cc:503] Master 127.27.84.254:41247 was elected leader, sending a full tablet report...
I20260808 15:52:45.772017 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.757557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.27.84.254:41247)
0808 15:52:45.757774 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.757781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.757799 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:52:45.757974 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.757979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.758052 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.758170 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:45.759812 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.760791 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.762308 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.762819 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.762969 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.765858 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:45.765863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:45.765866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:45.765902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:45.768810 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.769679 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.769685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.769690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.769856 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:45.770323 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:45.770328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:45.770330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:45.770759 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:45.771056 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.771397 (+ 341us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.771649 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260808 15:52:45.775857 20653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:52:45.783665 20653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:52:45.812623 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.799523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.27.84.194:37763)
0808 15:52:45.800081 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.800090 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.804124 (+ 4034us) server_negotiation.cc:407] Connection header received
0808 15:52:45.804521 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.804532 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.804767 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.804950 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:45.805898 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.807425 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.809617 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.811112 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.811256 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.811478 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.812073 (+ 595us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.812262 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":319,"threads_started":1}
I20260808 15:52:45.812659 21361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.796467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45697 (local address 127.0.0.1:53880)
0808 15:52:45.797698 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.797760 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.797998 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.804206 (+ 6208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.804210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.804270 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:45.804910 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.804927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.806939 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.806962 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.809491 (+ 2529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.809508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.811492 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.811499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.811712 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.811780 (+ 68us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.811920 (+ 140us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.812135 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":123,"spinlock_wait_cycles":16512,"thread_start_us":2413,"threads_started":1}
I20260808 15:52:45.813405 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.798838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37763 (local address 127.0.0.1:53518)
0808 15:52:45.803994 (+ 5156us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.804013 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.804265 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.805234 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.805241 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.805271 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:45.805702 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.805715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.807588 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.807593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.809455 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.809468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.811650 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.811657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.811837 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.811872 (+ 35us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.812006 (+ 134us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.812149 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5005,"mutex_wait_us":84,"spinlock_wait_cycles":768,"thread_start_us":4852,"threads_started":1}
I20260808 15:52:45.813570 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.796820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37734)
0808 15:52:45.799182 (+ 2362us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.799201 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.799320 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.803980 (+ 4660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.803990 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.804027 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:45.804721 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.804735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.806677 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.806682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.809491 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.809508 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.811876 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.811884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.812065 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.812096 (+ 31us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.812179 (+ 83us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.812310 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2205,"mutex_wait_us":85,"spinlock_wait_cycles":2944,"thread_start_us":986,"threads_started":1}
I20260808 15:52:45.814028 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.796801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.27.84.195:45697)
0808 15:52:45.803583 (+ 6782us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.803590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.803621 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:52:45.803725 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.803732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.803962 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.805091 (+ 1129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:45.805316 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.806775 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.809686 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.811220 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.812281 (+ 1061us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.812524 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.812608 (+ 84us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.813644 (+ 1036us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6620,"thread_start_us":6451,"threads_started":1}
I20260808 15:52:45.814417 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.797557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.27.84.193:40679)
0808 15:52:45.803204 (+ 5647us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.803214 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.803251 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:52:45.803370 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.803380 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.803673 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.804806 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:45.805194 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.806500 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.809689 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.811220 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.811443 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.811680 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.813904 (+ 2224us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.814083 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5456,"thread_start_us":5287,"threads_started":1}
I20260808 15:52:45.817783 21256 tablet_service.cc:1467] Processing CreateTablet for tablet 97b9be94727041eab8980407eb63588b (DEFAULT_TABLE table=test-table [id=9112b5675a2b42af8c28218a5e58f989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:52:45.817783 21045 tablet_service.cc:1467] Processing CreateTablet for tablet 97b9be94727041eab8980407eb63588b (DEFAULT_TABLE table=test-table [id=9112b5675a2b42af8c28218a5e58f989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:52:45.818127 20840 tablet_service.cc:1467] Processing CreateTablet for tablet 97b9be94727041eab8980407eb63588b (DEFAULT_TABLE table=test-table [id=9112b5675a2b42af8c28218a5e58f989]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:52:45.819401 20840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97b9be94727041eab8980407eb63588b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:52:45.819391 21256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97b9be94727041eab8980407eb63588b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:52:45.819415 21045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97b9be94727041eab8980407eb63588b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:52:45.832587 21369 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Bootstrap starting.
I20260808 15:52:45.836308 21369 tablet_bootstrap.cc:654] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:45.837579 21369 log.cc:824] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:45.840319 21370 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Bootstrap starting.
I20260808 15:52:45.840888 21369 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: No bootstrap required, opened a new log
I20260808 15:52:45.841179 21369 ts_tablet_manager.cc:1397] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260808 15:52:45.842769 21371 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Bootstrap starting.
I20260808 15:52:45.843585 21370 tablet_bootstrap.cc:654] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:45.844946 21370 log.cc:824] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:45.846475 21371 tablet_bootstrap.cc:654] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Neither blocks nor log segments found. Creating new log.
I20260808 15:52:45.847939 21371 log.cc:824] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:52:45.848637 21370 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: No bootstrap required, opened a new log
I20260808 15:52:45.848976 21370 ts_tablet_manager.cc:1397] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:52:45.850289 21369 raft_consensus.cc:348] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.850823 21369 raft_consensus.cc:374] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:45.850932 21369 raft_consensus.cc:729] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa94237026f46e9b482864485b310b5, State: Initialized, Role: FOLLOWER
I20260808 15:52:45.851596 21369 consensus_queue.cc:260] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.853262 21369 ts_tablet_manager.cc:1428] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260808 15:52:45.858444 21371 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: No bootstrap required, opened a new log
I20260808 15:52:45.858822 21371 ts_tablet_manager.cc:1397] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20260808 15:52:45.860172 21370 raft_consensus.cc:348] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.860579 21370 raft_consensus.cc:374] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:45.860702 21370 raft_consensus.cc:729] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633727d132ce4612b5fabbdca0ef1056, State: Initialized, Role: FOLLOWER
I20260808 15:52:45.861295 21370 consensus_queue.cc:260] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.862493 21346 heartbeater.cc:503] Master 127.27.84.254:41247 was elected leader, sending a full tablet report...
I20260808 15:52:45.863066 21370 ts_tablet_manager.cc:1428] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Time spent starting tablet: real 0.014s user 0.007s sys 0.009s
I20260808 15:52:45.868268 21371 raft_consensus.cc:348] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.868726 21371 raft_consensus.cc:374] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:52:45.868860 21371 raft_consensus.cc:729] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb3e9e0e980438d8eac29ab45bdd79b, State: Initialized, Role: FOLLOWER
I20260808 15:52:45.869510 21371 consensus_queue.cc:260] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.870707 21152 heartbeater.cc:503] Master 127.27.84.254:41247 was elected leader, sending a full tablet report...
I20260808 15:52:45.871215 21371 ts_tablet_manager.cc:1428] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260808 15:52:45.937839 21377 raft_consensus.cc:482] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:52:45.938149 21377 raft_consensus.cc:504] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.940088 21377 leader_election.cc:288] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679), 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
I20260808 15:52:45.947046 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.940620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:48051 (local address 127.27.84.193:40679)
0808 15:52:45.940857 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.940864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.941256 (+ 392us) server_negotiation.cc:407] Connection header received
0808 15:52:45.941504 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.941512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.941608 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.941714 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:45.942181 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.943187 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.945094 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.946042 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.946167 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.946219 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.946655 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.946795 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260808 15:52:45.947048 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.940578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.27.84.195:48051)
0808 15:52:45.941123 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.941148 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.941338 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.941751 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.941758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.941791 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:45.942020 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.942033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.943343 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.943348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.944924 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.944937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.946220 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.946227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.946434 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.946473 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.946587 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.946751 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":1}
I20260808 15:52:45.947623 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.940687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37763 (local address 127.27.84.195:48785)
0808 15:52:45.941505 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:45.941523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:52:45.941664 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:52:45.942054 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:52:45.942058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:52:45.942077 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:52:45.942515 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.942529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.943851 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.943855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.945373 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:52:45.945388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.946766 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.946772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:52:45.946961 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.946991 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 15:52:45.947099 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 15:52:45.947231 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":424,"threads_started":1}
I20260808 15:52:45.947918 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:45.940693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:48785 (local address 127.27.84.194:37763)
0808 15:52:45.940897 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:45.940903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:45.941634 (+ 731us) server_negotiation.cc:407] Connection header received
0808 15:52:45.941838 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:45.941846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:45.941929 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:45.942031 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:52:45.942788 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.943698 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.945542 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:45.946526 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:45.946716 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:45.946781 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:45.947134 (+ 353us) server_negotiation.cc:299] Negotiation successful
0808 15:52:45.947347 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:52:45.948459 20873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b9be94727041eab8980407eb63588b" candidate_uuid: "633727d132ce4612b5fabbdca0ef1056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa94237026f46e9b482864485b310b5" is_pre_election: true
I20260808 15:52:45.949100 20873 raft_consensus.cc:2454] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633727d132ce4612b5fabbdca0ef1056 in term 0.
I20260808 15:52:45.949030 21078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b9be94727041eab8980407eb63588b" candidate_uuid: "633727d132ce4612b5fabbdca0ef1056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" is_pre_election: true
I20260808 15:52:45.949556 21078 raft_consensus.cc:2454] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633727d132ce4612b5fabbdca0ef1056 in term 0.
I20260808 15:52:45.950034 21186 leader_election.cc:302] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa94237026f46e9b482864485b310b5, 633727d132ce4612b5fabbdca0ef1056; no voters:
I20260808 15:52:45.950552 21377 raft_consensus.cc:2781] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:52:45.950757 21377 raft_consensus.cc:482] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:52:45.950881 21377 raft_consensus.cc:3037] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:45.954758 21377 raft_consensus.cc:504] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:52:45.955657 21377 leader_election.cc:288] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [CANDIDATE]: Term 1 election: Requested vote from peers 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679), 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
W20260808 15:52:45.955873 21347 tablet.cc:2369] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:52:45.956434 20873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b9be94727041eab8980407eb63588b" candidate_uuid: "633727d132ce4612b5fabbdca0ef1056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa94237026f46e9b482864485b310b5"
I20260808 15:52:45.956662 20873 raft_consensus.cc:3037] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:45.956764 21078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97b9be94727041eab8980407eb63588b" candidate_uuid: "633727d132ce4612b5fabbdca0ef1056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b"
I20260808 15:52:45.957000 21078 raft_consensus.cc:3037] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:52:45.959774 20873 raft_consensus.cc:2454] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633727d132ce4612b5fabbdca0ef1056 in term 1.
I20260808 15:52:45.960193 21186 leader_election.cc:302] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa94237026f46e9b482864485b310b5, 633727d132ce4612b5fabbdca0ef1056; no voters:
I20260808 15:52:45.960256 21078 raft_consensus.cc:2454] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633727d132ce4612b5fabbdca0ef1056 in term 1.
I20260808 15:52:45.960544 21377 raft_consensus.cc:2781] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:52:45.961068 21377 raft_consensus.cc:686] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 1 LEADER]: Becoming Leader. State: Replica: 633727d132ce4612b5fabbdca0ef1056, State: Running, Role: LEADER
I20260808 15:52:45.961623 21377 consensus_queue.cc:237] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
W20260808 15:52:45.965924 21153 tablet.cc:2369] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:52:45.966284 20653 catalog_manager.cc:5495] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 reported cstate change: term changed from 0 to 1, leader changed from <none> to 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195). New cstate: current_term: 1 leader_uuid: "633727d132ce4612b5fabbdca0ef1056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } health_report { overall_health: UNKNOWN } } }
W20260808 15:52:45.969726 20947 tablet.cc:2369] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:52:46.009575 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.003721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53888 (local address 127.27.84.195:45697)
0808 15:52:46.003990 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:46.003997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:46.004113 (+ 116us) server_negotiation.cc:407] Connection header received
0808 15:52:46.004402 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:46.004407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:46.004491 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:46.004657 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:52:46.005084 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.005819 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.007300 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.007720 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.007861 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:46.008419 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:52:46.008961 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:52:46.009145 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:46.009211 (+ 66us) server_negotiation.cc:299] Negotiation successful
0808 15:52:46.009300 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:52:46.018555 21078 raft_consensus.cc:1261] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Refusing update from remote peer 633727d132ce4612b5fabbdca0ef1056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:52:46.018558 20873 raft_consensus.cc:1261] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Refusing update from remote peer 633727d132ce4612b5fabbdca0ef1056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:52:46.019686 21383 consensus_queue.cc:1035] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:52:46.020085 21377 consensus_queue.cc:1035] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:52:46.030807 21387 mvcc.cc:205] Tried to move back new op lower bound from 7316293083201105920 to 7316293082983157760. Current Snapshot: MvccSnapshot[applied={T|T < 7316293083201105920}]
I20260808 15:52:46.033493 21388 mvcc.cc:205] Tried to move back new op lower bound from 7316293083201105920 to 7316293082983157760. Current Snapshot: MvccSnapshot[applied={T|T < 7316293083201105920}]
I20260808 15:52:46.042528 21390 mvcc.cc:205] Tried to move back new op lower bound from 7316293083201105920 to 7316293082983157760. Current Snapshot: MvccSnapshot[applied={T|T < 7316293083201105920}]
I20260808 15:52:46.046348 20653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:52:46.046748 20653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:52:46.050981 20653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Sending DeleteTablet for 3 replicas of tablet 97b9be94727041eab8980407eb63588b
I20260808 15:52:46.052112 21045 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b9be94727041eab8980407eb63588b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:46 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:53518
I20260808 15:52:46.052081 21256 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b9be94727041eab8980407eb63588b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:46 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:53880
I20260808 15:52:46.052183 20840 tablet_service.cc:1514] Processing DeleteTablet for tablet 97b9be94727041eab8980407eb63588b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:52:46 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:37734
I20260808 15:52:46.053007 21402 tablet_replica.cc:330] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: stopping tablet replica
I20260808 15:52:46.053438 21402 raft_consensus.cc:2229] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:52:46.053687 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 20723
I20260808 15:52:46.053977 21402 raft_consensus.cc:2258] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:46.061642 21402 ts_tablet_manager.cc:1905] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:46.066475 21404 tablet_replica.cc:330] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: stopping tablet replica
I20260808 15:52:46.066994 21404 raft_consensus.cc:2229] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:52:46.067359 21403 tablet_replica.cc:330] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: stopping tablet replica
I20260808 15:52:46.067592 21404 raft_consensus.cc:2258] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:46.067802 21403 raft_consensus.cc:2229] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:52:46.068542 21403 raft_consensus.cc:2258] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:52:46.070865 21403 ts_tablet_manager.cc:1905] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:52:46.071583 21402 ts_tablet_manager.cc:1918] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:46.071774 21402 log.cc:1196] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/97b9be94727041eab8980407eb63588b
I20260808 15:52:46.072359 21402 ts_tablet_manager.cc:1939] T 97b9be94727041eab8980407eb63588b P 2fb3e9e0e980438d8eac29ab45bdd79b: Deleting consensus metadata
I20260808 15:52:46.072475 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.054807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.27.84.193:40679)
0808 15:52:46.054989 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:46.055000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:46.055015 (+ 15us) server_negotiation.cc:407] Connection header received
0808 15:52:46.055063 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:46.055066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:46.055124 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:46.055232 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:46.056702 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.057574 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.059150 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.059781 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.059954 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:46.063546 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:46.063562 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:46.063581 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:46.063622 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:46.067832 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:46.068940 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:46.068952 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:46.068955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:46.069127 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:46.069607 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:46.069612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:46.069615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:46.070095 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:46.070422 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:46.071724 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0808 15:52:46.072036 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:52:46.073238 20937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:46.074355 20635 catalog_manager.cc:4841] TS 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763): tablet 97b9be94727041eab8980407eb63588b (table test-table [id=9112b5675a2b42af8c28218a5e58f989]) successfully deleted
I20260808 15:52:46.084015 21403 ts_tablet_manager.cc:1918] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:52:46.084251 21403 log.cc:1196] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/97b9be94727041eab8980407eb63588b
I20260808 15:52:46.084872 21403 ts_tablet_manager.cc:1939] T 97b9be94727041eab8980407eb63588b P 633727d132ce4612b5fabbdca0ef1056: Deleting consensus metadata
I20260808 15:52:46.086936 20632 catalog_manager.cc:4841] TS 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697): tablet 97b9be94727041eab8980407eb63588b (table test-table [id=9112b5675a2b42af8c28218a5e58f989]) successfully deleted
I20260808 15:52:46.166464 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 20723
W20260808 15:52:46.195050 20631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:40679 (error 108)
I20260808 15:52:46.195441 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 20951
I20260808 15:52:46.197337 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.196919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37742)
0808 15:52:46.197093 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.197199 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 15:52:46.197608 20631 catalog_manager.cc:4584] TS 2aa94237026f46e9b482864485b310b5 (127.27.84.193:40679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97b9be94727041eab8980407eb63588b: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
I20260808 15:52:46.212563 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.197003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53522 (local address 127.27.84.194:37763)
0808 15:52:46.197196 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:46.197202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:46.197222 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:52:46.197290 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:46.197296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:46.197388 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:46.197503 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:46.199602 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.200346 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.202293 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.202745 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.202845 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:46.206181 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:46.206197 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:46.206212 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:46.206246 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:46.209174 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:46.209991 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:46.209998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:46.210001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:46.210121 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:46.210532 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:46.210535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:46.210537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:46.211019 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:46.211318 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:46.211781 (+ 463us) server_negotiation.cc:299] Negotiation successful
0808 15:52:46.212026 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:52:46.213376 21143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:46.251618 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.251027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37758)
0808 15:52:46.251284 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.251422 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260808 15:52:46.252995 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.252531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37768)
0808 15:52:46.252741 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.252836 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:52:46.306166 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 20951
I20260808 15:52:46.313943 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.313370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37782)
0808 15:52:46.313588 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.313707 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:52:46.315053 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.314711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37786)
0808 15:52:46.314861 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.314942 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260808 15:52:46.338510 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 21155
I20260808 15:52:46.354161 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.339440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.27.84.195:45697)
0808 15:52:46.339706 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0808 15:52:46.339712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:52:46.339733 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:52:46.339908 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:52:46.339918 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:52:46.339988 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:52:46.340123 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:52:46.341497 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.342345 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.343983 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:52:46.344453 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:52:46.344612 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:52:46.347678 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:52:46.347692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:52:46.347706 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:52:46.347743 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:52:46.351119 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:46.351888 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:46.351905 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:46.351907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:46.352086 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:52:46.352485 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:52:46.352491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:52:46.352494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:52:46.352861 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:52:46.353220 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0808 15:52:46.353526 (+ 306us) server_negotiation.cc:299] Negotiation successful
0808 15:52:46.353760 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260808 15:52:46.355057 21338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:46.395812 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.395181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37800)
0808 15:52:46.395463 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.395582 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260808 15:52:46.397008 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:52:46.396569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40679 (local address 127.0.0.1:37802)
0808 15:52:46.396786 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0808 15:52:46.396878 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:40679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260808 15:52:46.480238 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 21155
I20260808 15:52:46.504539 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 20609
I20260808 15:52:46.505856 20712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:52:46.621083 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 20609
I20260808 15:53:06.657465 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41247
--webserver_interface=127.27.84.254
--webserver_port=40731
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:41247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:07.017968 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:07.018291 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:07.018472 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:07.025975 21410 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260808 15:53:07.026099 21410 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260808 15:53:07.026176 21410 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260808 15:53:07.026253 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:07.026353 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:07.026484 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:07.026546 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:07.026667 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:07.036847 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:41247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=40731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:07.038244 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:07.039728 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:07.052590 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:07.052567 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:07.052520 21423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:07.053381 21410 server_base.cc:1034] running on GCE node
I20260808 15:53:07.054224 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:07.055717 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:07.056948 21410 hybrid_clock.cc:648] HybridClock initialized: now 1786204387056885 us; error 32 us; skew 500 ppm
I20260808 15:53:07.057291 21410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:07 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204387, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:07.060765 21410 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:53:07.062220 21410 webserver.cc:460] Webserver started at http://127.27.84.254:40731/ using document root <none> and password file <none>
I20260808 15:53:07.063012 21410 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:07.063143 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:07.068636 21410 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:53:07.071763 21434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:07.073235 21410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:53:07.073446 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "47f7304d0fea46a69e5de64fe9e80943"
format_stamp: "Formatted at 2026-08-08 15:52:20 on dist-test-slave-twwt"
server_key: "a8037df5862334a09d106555b3e75cf8"
server_key_iv: "76a94c09f8d0c11d5cdad738f676cb3c"
server_key_version: "encryptionkey@0"
I20260808 15:53:07.074473 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:07.096745 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:07.101361 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:07.101679 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:07.121835 21410 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:41247
I20260808 15:53:07.121929 21509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:41247 every 8 connection(s)
I20260808 15:53:07.123404 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:07.130405 21510 sys_catalog.cc:263] Verifying existing consensus state
I20260808 15:53:07.131186 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 21410
I20260808 15:53:07.132120 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:40679
--local_ip_for_outbound_sockets=127.27.84.193
--tserver_master_addrs=127.27.84.254:41247
--webserver_port=39071
--webserver_interface=127.27.84.193
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:53:07.133874 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap starting.
I20260808 15:53:07.155025 21510 log.cc:824] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:07.172048 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:53:07.172654 21510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Bootstrap complete.
I20260808 15:53:07.182812 21510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:53:07.183799 21510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f7304d0fea46a69e5de64fe9e80943, State: Initialized, Role: FOLLOWER
I20260808 15:53:07.184544 21510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:53:07.184804 21510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:07.184979 21510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:07.185170 21510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 2 FOLLOWER]: Advancing to term 3
I20260808 15:53:07.189355 21510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:53:07.189852 21510 leader_election.cc:302] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47f7304d0fea46a69e5de64fe9e80943; no voters:
I20260808 15:53:07.190563 21510 leader_election.cc:288] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260808 15:53:07.190794 21516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 3 FOLLOWER]: Leader election won for term 3
I20260808 15:53:07.192101 21516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [term 3 LEADER]: Becoming Leader. State: Replica: 47f7304d0fea46a69e5de64fe9e80943, State: Running, Role: LEADER
I20260808 15:53:07.192643 21516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } }
I20260808 15:53:07.193017 21510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:07.195434 21518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47f7304d0fea46a69e5de64fe9e80943. Latest consensus state: current_term: 3 leader_uuid: "47f7304d0fea46a69e5de64fe9e80943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } } }
I20260808 15:53:07.195868 21518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:07.196380 21517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "47f7304d0fea46a69e5de64fe9e80943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47f7304d0fea46a69e5de64fe9e80943" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41247 } } }
I20260808 15:53:07.196674 21517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:07.201527 21523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:07.212431 21523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:07.214459 21523 catalog_manager.cc:1259] Loaded cluster ID: 7b267f7c22524a1586d979a359603a56
I20260808 15:53:07.214586 21523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:07.221743 21523 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:07.223767 21523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Loaded TSK: 0
I20260808 15:53:07.223937 21523 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 47f7304d0fea46a69e5de64fe9e80943: Loaded TSK: 1
F20260808 15:53:07.225342 21523 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff3716499ad google::LogMessage::Fail() at ??:0
@ 0x7ff37164c88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff3716494de google::LogMessage::Flush() at ??:0
@ 0x7ff37164aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff3753ad7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff37539c73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff3870781c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff387077783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff3871032a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff3871030c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff38714d5e9 std::function<>::operator()() at ??:0
@ 0x7ff38707bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff387079c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff387101610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff387101439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff374295345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff37429891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff374298799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff374273eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff37754d6db start_thread at ??:0
@ 0x7ff3704e361f clone at ??:0
W20260808 15:53:07.548024 21514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:07.548375 21514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:07.548568 21514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:07.556190 21514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:07.556358 21514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:07.556543 21514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:07.567047 21514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:40679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=39071
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:07.568492 21514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:07.569917 21514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:07.583889 21582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:07.584007 21584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:07.583891 21580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:07.585146 21514 server_base.cc:1034] running on GCE node
I20260808 15:53:07.585846 21514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:07.586947 21514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:07.588205 21514 hybrid_clock.cc:648] HybridClock initialized: now 1786204387588152 us; error 49 us; skew 500 ppm
I20260808 15:53:07.588553 21514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:07 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204387, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:07.592185 21514 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:53:07.593815 21514 webserver.cc:460] Webserver started at http://127.27.84.193:39071/ using document root <none> and password file <none>
I20260808 15:53:07.594533 21514 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:07.594722 21514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:07.600445 21514 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:53:07.603513 21592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:07.605036 21514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:53:07.605216 21514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "2aa94237026f46e9b482864485b310b5"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "ecdf03102f35112172df703d005e190a"
server_key_iv: "d8eaa810d666c9bcb2f220311ae0f554"
server_key_version: "encryptionkey@0"
I20260808 15:53:07.606248 21514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:07.623642 21514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:07.627954 21514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:07.628249 21514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:07.629326 21514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:07.631279 21601 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260808 15:53:07.637133 21514 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260808 15:53:07.637297 21514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260808 15:53:07.637470 21514 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260808 15:53:07.640885 21514 ts_tablet_manager.cc:610] Registered 1 tablets
I20260808 15:53:07.640972 21514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260808 15:53:07.641341 21601 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Bootstrap starting.
I20260808 15:53:07.650393 21601 log.cc:824] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:07.667122 21601 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260808 15:53:07.667838 21601 tablet_bootstrap.cc:492] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Bootstrap complete.
I20260808 15:53:07.668324 21601 ts_tablet_manager.cc:1397] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.008s
I20260808 15:53:07.677858 21601 raft_consensus.cc:348] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:53:07.679771 21601 raft_consensus.cc:729] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa94237026f46e9b482864485b310b5, State: Initialized, Role: FOLLOWER
I20260808 15:53:07.680540 21601 consensus_queue.cc:260] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:53:07.682128 21601 ts_tablet_manager.cc:1428] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260808 15:53:07.688711 21514 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:40679
I20260808 15:53:07.688812 21764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:40679 every 8 connection(s)
I20260808 15:53:07.690016 21514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:07.694922 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:07.692894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:59589)
0808 15:53:07.693709 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:07.693935 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":328,"threads_started":1}
W20260808 15:53:07.695652 21765 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:41247 (0 consecutive failures): Network error: Failed to ping master at 127.27.84.254:41247: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
I20260808 15:53:07.696624 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:07.696187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:54665)
0808 15:53:07.696377 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:07.696476 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 15:53:07.696952 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 21514
I20260808 15:53:07.697547 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:07.697218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:41557)
0808 15:53:07.697358 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:07.697430 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260808 15:53:07.697781 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:37763
--local_ip_for_outbound_sockets=127.27.84.194
--tserver_master_addrs=127.27.84.254:41247
--webserver_port=36305
--webserver_interface=127.27.84.194
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:07.697917 21765 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:53:08.054739 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:08.055052 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:08.055244 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:08.062949 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:08.063072 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:08.063282 21768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:08.073341 21768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:37763
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=36305
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:08.074760 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:08.076161 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:08.090224 21776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:08.090277 21779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:08.090301 21775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:08.091542 21768 server_base.cc:1034] running on GCE node
I20260808 15:53:08.092216 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:08.093339 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:08.094591 21768 hybrid_clock.cc:648] HybridClock initialized: now 1786204388094531 us; error 54 us; skew 500 ppm
I20260808 15:53:08.095067 21768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:08 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204388, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:08.098263 21768 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:53:08.099761 21768 webserver.cc:460] Webserver started at http://127.27.84.194:36305/ using document root <none> and password file <none>
I20260808 15:53:08.100435 21768 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:08.100541 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:08.106055 21768 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260808 15:53:08.108927 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:08.110427 21768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:53:08.110560 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "2fb3e9e0e980438d8eac29ab45bdd79b"
format_stamp: "Formatted at 2026-08-08 15:52:21 on dist-test-slave-twwt"
server_key: "d26b3dedb35d016b37548625b83aa343"
server_key_iv: "a5f11bcb429af6c12af2ea74e61f24db"
server_key_version: "encryptionkey@0"
I20260808 15:53:08.111726 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:08.136328 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:08.140509 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:08.140767 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:08.141824 21768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:08.143211 21768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:08.143303 21768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:08.143385 21768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:08.143442 21768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:08.178853 21768 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:37763
I20260808 15:53:08.179045 21958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:37763 every 8 connection(s)
I20260808 15:53:08.180408 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:08.182642 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 21768
I20260808 15:53:08.183446 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:45697
--local_ip_for_outbound_sockets=127.27.84.195
--tserver_master_addrs=127.27.84.254:41247
--webserver_port=33939
--webserver_interface=127.27.84.195
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:53:08.186133 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.184115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:58053)
0808 15:53:08.184958 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.185221 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":331,"threads_started":1}
W20260808 15:53:08.186986 21960 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:41247 (0 consecutive failures): Network error: Failed to ping master at 127.27.84.254:41247: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
I20260808 15:53:08.187937 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.187579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:45629)
0808 15:53:08.187739 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.187816 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260808 15:53:08.188874 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.188532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:39739)
0808 15:53:08.188676 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.188757 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260808 15:53:08.189177 21960 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260808 15:53:08.536850 21963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:08.537145 21963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:08.537313 21963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:08.544564 21963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:08.544682 21963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:08.544868 21963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:08.555228 21963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:45697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=33939
--tserver_master_addrs=127.27.84.254:41247
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:08.556612 21963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:08.558014 21963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:08.572366 21968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:08.572342 21970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:08.572400 21975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:08.573897 21963 server_base.cc:1034] running on GCE node
I20260808 15:53:08.574539 21963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:08.575646 21963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:08.576900 21963 hybrid_clock.cc:648] HybridClock initialized: now 1786204388576825 us; error 66 us; skew 500 ppm
I20260808 15:53:08.577265 21963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:08 dist-test-slave-twwt krb5kdc[19772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204388, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:08.580554 21963 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:53:08.581986 21963 webserver.cc:460] Webserver started at http://127.27.84.195:33939/ using document root <none> and password file <none>
I20260808 15:53:08.582712 21963 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:08.582829 21963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:08.588418 21963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260808 15:53:08.591213 21984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:08.592607 21963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:53:08.592788 21963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "633727d132ce4612b5fabbdca0ef1056"
format_stamp: "Formatted at 2026-08-08 15:52:22 on dist-test-slave-twwt"
server_key: "41daf653703b70e94c371688abff05c2"
server_key_iv: "d0df808767472e442ffd454e48f536fc"
server_key_version: "encryptionkey@0"
I20260808 15:53:08.593734 21963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:08.614212 21963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:08.618767 21963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:08.619011 21963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:08.619982 21963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:08.621254 21963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:08.621351 21963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:08.621433 21963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:08.621496 21963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:08.656363 21963 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:45697
I20260808 15:53:08.656477 22142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:45697 every 8 connection(s)
I20260808 15:53:08.657896 21963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:08.663007 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.660963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:46069)
0808 15:53:08.661685 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.661928 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":282,"threads_started":1}
W20260808 15:53:08.663731 22143 heartbeater.cc:644] Failed to heartbeat to 127.27.84.254:41247 (0 consecutive failures): Network error: Failed to ping master at 127.27.84.254:41247: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
I20260808 15:53:08.664525 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.664196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:57069)
0808 15:53:08.664328 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.664408 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260808 15:53:08.665354 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.665067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:55737)
0808 15:53:08.665180 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.665251 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260808 15:53:08.665650 22143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260808 15:53:08.667807 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 21963
I20260808 15:53:08.699304 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:08.698459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:45733)
0808 15:53:08.698987 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:08.699072 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":326,"threads_started":1}
I20260808 15:53:09.190563 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.189746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:34567)
0808 15:53:09.190223 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:09.190313 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":245,"threads_started":1}
I20260808 15:53:09.394596 22152 raft_consensus.cc:482] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:09.394973 22152 raft_consensus.cc:504] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:53:09.396615 22152 leader_election.cc:288] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697), 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
Aug 08 15:53:09 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204387, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
Aug 08 15:53:09 dist-test-slave-twwt krb5kdc[19772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204387, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:53:09.434190 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.396973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:37763 (local address 127.27.84.193:52965)
0808 15:53:09.407423 (+ 10450us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:09.407499 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:09.408245 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:09.411904 (+ 3659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:09.411917 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:09.413200 (+ 1283us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:09.414548 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:09.414581 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.417077 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.417082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:09.419738 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:09.419757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.421363 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.421375 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:09.421789 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:09.422868 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:09.422912 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:09.426546 (+ 3634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:09.430344 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:09.430348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:09.430351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:09.430892 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:09.431427 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:09.431430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:09.431436 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:09.431594 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:09.432506 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:09.432515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:09.432845 (+ 330us) client_negotiation.cc:769] Sending connection context
0808 15:53:09.433121 (+ 276us) client_negotiation.cc:240] Negotiation successful
0808 15:53:09.433711 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10286,"mutex_wait_us":451,"spinlock_wait_cycles":33664,"thread_start_us":11238,"threads_started":1}
I20260808 15:53:09.434180 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.397267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:43633 (local address 127.27.84.195:45697)
0808 15:53:09.407862 (+ 10595us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:09.407879 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:09.408486 (+ 607us) server_negotiation.cc:407] Connection header received
0808 15:53:09.410751 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:09.410762 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:09.411307 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:09.411716 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:09.414830 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.416830 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.420059 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.421121 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.421348 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:09.426663 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:09.426677 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:09.426702 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:09.426763 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:09.430042 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:09.431044 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:09.431051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:09.431057 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:09.431238 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:09.431818 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:09.431830 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:09.431833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:09.432349 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:09.432559 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:09.433348 (+ 789us) server_negotiation.cc:299] Negotiation successful
0808 15:53:09.433712 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10387,"thread_start_us":10268,"threads_started":1}
I20260808 15:53:09.434190 22156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.397356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:52965 (local address 127.27.84.194:37763)
0808 15:53:09.408360 (+ 11004us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:09.408377 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:09.408442 (+ 65us) server_negotiation.cc:407] Connection header received
0808 15:53:09.410882 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:09.410892 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:09.411307 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:09.411715 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:09.414850 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.416839 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.419979 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.421122 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.421348 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:09.426962 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:09.426974 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:09.426993 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:09.427043 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:09.430193 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:09.431080 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:09.431087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:09.431094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:09.431264 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:09.431836 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:09.431840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:09.431842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:09.432351 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:09.432568 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:09.433368 (+ 800us) server_negotiation.cc:299] Negotiation successful
0808 15:53:09.433772 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10820,"thread_start_us":10641,"threads_started":1}
I20260808 15:53:09.434679 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.397087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:45697 (local address 127.27.84.193:43633)
0808 15:53:09.408327 (+ 11240us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:09.408355 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:09.408478 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:09.411904 (+ 3426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:09.411917 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:09.413200 (+ 1283us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:09.414548 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:09.414581 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.417075 (+ 2494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.417082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:09.419738 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:09.419757 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:09.421396 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:09.421403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:09.421791 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:09.422868 (+ 1077us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:09.422911 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:09.426224 (+ 3313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:09.430213 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:09.430223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:09.430239 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:09.430879 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:09.431365 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:09.431374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:09.431377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:09.431591 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:09.432506 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:09.432515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:09.432847 (+ 332us) client_negotiation.cc:769] Sending connection context
0808 15:53:09.433123 (+ 276us) client_negotiation.cc:240] Negotiation successful
0808 15:53:09.433894 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11099,"mutex_wait_us":772,"spinlock_wait_cycles":47744,"thread_start_us":10030,"threads_started":1}
W20260808 15:53:09.437525 21598 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
W20260808 15:53:09.437784 21596 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
I20260808 15:53:09.437919 21596 leader_election.cc:302] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa94237026f46e9b482864485b310b5; no voters: 2fb3e9e0e980438d8eac29ab45bdd79b, 633727d132ce4612b5fabbdca0ef1056
I20260808 15:53:09.438338 22152 raft_consensus.cc:2726] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:53:09.667613 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.666297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:49117)
0808 15:53:09.667001 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:09.667215 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":350,"threads_started":1}
I20260808 15:53:09.700624 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:09.700173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:36467)
0808 15:53:09.700387 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:09.700487 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260808 15:53:10.192340 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:10.191573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:34469)
0808 15:53:10.192071 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:10.192162 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260808 15:53:10.669262 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:10.668572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:49601)
0808 15:53:10.669020 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:10.669104 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260808 15:53:10.702302 22160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:10.701566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:59033)
0808 15:53:10.702026 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:10.702113 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":238,"threads_started":1}
I20260808 15:53:11.194226 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:11.193393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:38351)
0808 15:53:11.193942 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:11.194037 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
I20260808 15:53:11.528398 22162 raft_consensus.cc:482] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:11.528609 22162 raft_consensus.cc:504] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:53:11.529340 22162 leader_election.cc:288] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697), 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
W20260808 15:53:11.530462 21598 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
W20260808 15:53:11.530681 21596 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
I20260808 15:53:11.530776 21596 leader_election.cc:302] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa94237026f46e9b482864485b310b5; no voters: 2fb3e9e0e980438d8eac29ab45bdd79b, 633727d132ce4612b5fabbdca0ef1056
I20260808 15:53:11.531005 22162 raft_consensus.cc:2726] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:53:11.670933 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:11.670188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:45621)
0808 15:53:11.670684 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:11.670769 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":234,"threads_started":1}
I20260808 15:53:11.704097 22164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:11.703297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:40081)
0808 15:53:11.703814 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:11.703908 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
I20260808 15:53:12.195997 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:12.195242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:39177)
0808 15:53:12.195707 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:12.195793 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260808 15:53:12.672868 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:12.672017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:34561)
0808 15:53:12.672554 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:12.672660 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":289,"threads_started":1}
I20260808 15:53:12.706002 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:12.705028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:58791)
0808 15:53:12.705636 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:12.705759 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260808 15:53:13.197926 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:13.197026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:59057)
0808 15:53:13.197641 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:13.197750 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":325,"threads_started":1}
I20260808 15:53:13.659375 22172 raft_consensus.cc:482] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:13.659584 22172 raft_consensus.cc:504] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:53:13.660302 22172 leader_election.cc:288] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697), 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
W20260808 15:53:13.661578 21598 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
W20260808 15:53:13.661784 21596 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
I20260808 15:53:13.661887 21596 leader_election.cc:302] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa94237026f46e9b482864485b310b5; no voters: 2fb3e9e0e980438d8eac29ab45bdd79b, 633727d132ce4612b5fabbdca0ef1056
I20260808 15:53:13.662122 22172 raft_consensus.cc:2726] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:53:13.674645 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:13.673882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:46669)
0808 15:53:13.674314 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:13.674407 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260808 15:53:13.707767 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:13.707001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:51833)
0808 15:53:13.707470 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:13.707566 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":232,"threads_started":1}
I20260808 15:53:14.199628 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:14.198905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:44619)
0808 15:53:14.199359 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:14.199440 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
I20260808 15:53:14.676604 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:14.675631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:51741)
0808 15:53:14.676242 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:14.676351 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":322,"threads_started":1}
I20260808 15:53:14.709482 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:14.708720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:37731)
0808 15:53:14.709163 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:14.709263 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20260808 15:53:15.201442 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:15.200582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:59959)
0808 15:53:15.201122 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:15.201220 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":244,"threads_started":1}
I20260808 15:53:15.678781 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:15.677694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:36831)
0808 15:53:15.678394 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:15.678500 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":325,"threads_started":1}
I20260808 15:53:15.711323 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:15.710505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:35549)
0808 15:53:15.710997 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:15.711091 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":285,"threads_started":1}
I20260808 15:53:16.203421 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:16.202416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:60857)
0808 15:53:16.203090 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:16.203229 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":397,"threads_started":1}
I20260808 15:53:16.680488 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:16.679762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:45607)
0808 15:53:16.680199 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:16.680281 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260808 15:53:16.712913 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:16.712153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:57115)
0808 15:53:16.712619 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:16.712719 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":237,"threads_started":1}
I20260808 15:53:17.205529 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:17.204567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:46903)
0808 15:53:17.205167 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:17.205263 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260808 15:53:17.631138 22192 raft_consensus.cc:482] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:17.631352 22192 raft_consensus.cc:504] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "633727d132ce4612b5fabbdca0ef1056" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 45697 } } peers { permanent_uuid: "2aa94237026f46e9b482864485b310b5" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40679 } } peers { permanent_uuid: "2fb3e9e0e980438d8eac29ab45bdd79b" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 37763 } }
I20260808 15:53:17.632001 22192 leader_election.cc:288] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697), 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763)
W20260808 15:53:17.633114 21598 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2fb3e9e0e980438d8eac29ab45bdd79b (127.27.84.194:37763): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
W20260808 15:53:17.633349 21596 leader_election.cc:341] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633727d132ce4612b5fabbdca0ef1056 (127.27.84.195:45697): Not found: Tablet not found: 97b9be94727041eab8980407eb63588b
I20260808 15:53:17.633535 21596 leader_election.cc:302] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2aa94237026f46e9b482864485b310b5; no voters: 2fb3e9e0e980438d8eac29ab45bdd79b, 633727d132ce4612b5fabbdca0ef1056
I20260808 15:53:17.633811 22192 raft_consensus.cc:2726] T 97b9be94727041eab8980407eb63588b P 2aa94237026f46e9b482864485b310b5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260808 15:53:17.682325 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:17.681531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:37721)
0808 15:53:17.682067 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:17.682160 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":250,"threads_started":1}
I20260808 15:53:17.714661 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:17.713833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:45895)
0808 15:53:17.714324 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:17.714415 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":245,"threads_started":1}
I20260808 15:53:18.207428 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:18.206570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:33783)
0808 15:53:18.207129 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:18.207223 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":285,"threads_started":1}
I20260808 15:53:18.684096 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:18.683364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:58375)
0808 15:53:18.683827 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:18.683911 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20260808 15:53:18.716434 22198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:18.715598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:54975)
0808 15:53:18.716118 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:18.716239 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260808 15:53:19.209116 22199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:19.208365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:39753)
0808 15:53:19.208847 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:19.208944 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":245,"threads_started":1}
I20260808 15:53:19.685789 22200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:19.685042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:32895)
0808 15:53:19.685533 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:19.685618 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":262,"threads_started":1}
I20260808 15:53:19.718135 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:19.717456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:50359)
0808 15:53:19.717885 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:19.717965 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":220,"threads_started":1}
I20260808 15:53:20.210913 22202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:20.210049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:57485)
0808 15:53:20.210495 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:20.210662 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":234,"threads_started":1}
I20260808 15:53:20.687671 22204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:20.686942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:45675)
0808 15:53:20.687407 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:20.687488 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260808 15:53:20.720096 22205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:20.719164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:33429)
0808 15:53:20.719739 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:20.719862 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":330,"threads_started":1}
I20260808 15:53:21.212860 22206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:21.211915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:38569)
0808 15:53:21.212527 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:21.212671 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":263,"threads_started":1}
I20260808 15:53:21.689487 22209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:21.688681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:53859)
0808 15:53:21.689192 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:21.689299 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":247,"threads_started":1}
I20260808 15:53:21.721942 22211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:21.721011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:40885)
0808 15:53:21.721616 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:21.721742 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":324,"threads_started":1}
I20260808 15:53:22.214740 22212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:22.213838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:45829)
0808 15:53:22.214390 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:22.214476 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":285,"threads_started":1}
I20260808 15:53:22.691383 22214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:22.690465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:55169)
0808 15:53:22.691016 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:22.691119 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260808 15:53:22.724637 22215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:22.723239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:48141)
0808 15:53:22.724347 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:22.724452 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":974,"thread_start_us":797,"threads_started":1}
I20260808 15:53:23.216784 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:23.216042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.194:47139)
0808 15:53:23.216510 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:23.216610 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":221,"threads_started":1}
I20260808 15:53:23.672159 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 21514
I20260808 15:53:23.691852 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:23.673700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.27.84.193:40679)
0808 15:53:23.674328 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:23.674340 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:23.674378 (+ 38us) server_negotiation.cc:407] Connection header received
0808 15:53:23.674465 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:23.674472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:23.674788 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:23.674943 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:23.676339 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:23.677320 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:23.678691 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:23.679281 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:23.679391 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:23.685772 (+ 6381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:23.685786 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:23.685802 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:23.685841 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:23.688736 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:23.689450 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:23.689457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:23.689459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:23.689589 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:23.690143 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:23.690146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:23.690148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:23.690550 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:23.690845 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:23.691245 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 15:53:23.691449 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":283,"threads_started":1}
I20260808 15:53:23.693419 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:23.692598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.195:48763)
0808 15:53:23.693116 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:23.693226 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260808 15:53:23.693490 21759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:23.786835 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:23.785584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41247 (local address 127.27.84.193:55907)
0808 15:53:23.786240 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:23.786436 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.254:41247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"threads_started":1}
I20260808 15:53:23.787655 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 21514
I20260808 15:53:23.816097 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 21768
I20260808 15:53:23.830778 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:23.816928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.27.84.194:37763)
0808 15:53:23.817475 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:23.817480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:23.817497 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:53:23.817556 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:23.817559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:23.817620 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:23.817716 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:23.819324 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:23.820181 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:23.821483 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:23.821957 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:23.822061 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:23.824738 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:23.824743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:23.824746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:23.824780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:23.827897 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:23.828683 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:23.828688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:23.828690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:23.828881 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:23.829222 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:23.829225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:23.829228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:23.829531 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:23.829768 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:23.830037 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 15:53:23.830235 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
I20260808 15:53:23.831661 21951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:23.919426 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 21768
I20260808 15:53:23.946324 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 21963
I20260808 15:53:23.964236 22227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:23.947410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.27.84.195:45697)
0808 15:53:23.948030 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:23.948035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:23.948052 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:53:23.948113 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:23.948117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:23.948186 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:23.948326 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:23.949928 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:23.950962 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:23.953144 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:23.953924 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:23.954091 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:23.957972 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:23.957979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:23.957985 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:23.958028 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:23.961424 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:23.962116 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:23.962122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:23.962126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:23.962294 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:23.962720 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:23.962726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:23.962729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:23.963015 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:23.963308 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:23.963602 (+ 294us) server_negotiation.cc:299] Negotiation successful
0808 15:53:23.963825 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":326,"threads_started":1}
I20260808 15:53:23.964954 22136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:24.051021 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 21963
2026-08-08T15:53:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65543 ms)
[----------] 30 tests from SecurityITest (314325 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:24 dist-test-slave-twwt krb5kdc[22232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:24 dist-test-slave-twwt krb5kdc[22232](info): set up 2 sockets
Aug 08 15:53:24 dist-test-slave-twwt krb5kdc[22232](info): commencing operation
krb5kdc: starting...
W20260808 15:53:25.469007 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.364s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:25 dist-test-slave-twwt krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:53:25.815192 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34627
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:34627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:26.179077 22254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:26.179425 22254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:26.179598 22254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:26.186949 22254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:26.187063 22254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:26.187181 22254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:26.187279 22254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:26.187371 22254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:26.197472 22254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:34627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:26.198897 22254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:26.200299 22254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:26.212678 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:26.212644 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:26.212946 22264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:26.213219 22254 server_base.cc:1034] running on GCE node
I20260808 15:53:26.214118 22254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:26.215569 22254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:26.216806 22254 hybrid_clock.cc:648] HybridClock initialized: now 1786204406216762 us; error 48 us; skew 500 ppm
I20260808 15:53:26.217185 22254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:26 dist-test-slave-twwt krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:26.220463 22254 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:53:26.222028 22254 webserver.cc:460] Webserver started at http://127.27.84.254:46575/ using document root <none> and password file <none>
I20260808 15:53:26.222810 22254 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:26.222970 22254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:26.223289 22254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:26.226357 22254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "b55d5c4866e34119838be5d13f7da8c3"
format_stamp: "Formatted at 2026-08-08 15:53:26 on dist-test-slave-twwt"
server_key: "7b4d5b56eaae0881661d82e5fccfc42a"
server_key_iv: "624fbab4339121383a139b1ded023c5b"
server_key_version: "encryptionkey@0"
I20260808 15:53:26.227188 22254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "b55d5c4866e34119838be5d13f7da8c3"
format_stamp: "Formatted at 2026-08-08 15:53:26 on dist-test-slave-twwt"
server_key: "7b4d5b56eaae0881661d82e5fccfc42a"
server_key_iv: "624fbab4339121383a139b1ded023c5b"
server_key_version: "encryptionkey@0"
I20260808 15:53:26.232756 22254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260808 15:53:26.236590 22280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:26.238080 22254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:53:26.238286 22254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "b55d5c4866e34119838be5d13f7da8c3"
format_stamp: "Formatted at 2026-08-08 15:53:26 on dist-test-slave-twwt"
server_key: "7b4d5b56eaae0881661d82e5fccfc42a"
server_key_iv: "624fbab4339121383a139b1ded023c5b"
server_key_version: "encryptionkey@0"
I20260808 15:53:26.238524 22254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:26.278478 22254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:26.283130 22254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:26.283437 22254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:26.303387 22254 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:34627
I20260808 15:53:26.303434 22349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:34627 every 8 connection(s)
I20260808 15:53:26.304822 22254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:26.308750 22351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:26.311092 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 22254
I20260808 15:53:26.311313 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:26.311973 27987 external_mini_cluster.cc:1428] Setting key 5167717cc08422ab4c37a8cfd6e5ee00
I20260808 15:53:26.322250 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3: Bootstrap starting.
I20260808 15:53:26.326013 22351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:26.327122 22351 log.cc:824] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:53:26 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:26.330387 22351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3: No bootstrap required, opened a new log
I20260808 15:53:26.339241 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:26.314191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.27.84.254:34627)
0808 15:53:26.315070 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:26.315086 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:26.315137 (+ 51us) server_negotiation.cc:407] Connection header received
0808 15:53:26.317585 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:26.317618 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:26.318166 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:26.318649 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:26.320036 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:26.322033 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:26.323566 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:26.324696 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:26.324908 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:26.329680 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:26.329688 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:26.329703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:26.329745 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:26.334023 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:26.334870 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:26.334880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:26.334889 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:26.335122 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:26.335618 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:26.335623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:26.335626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:26.336397 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:26.336719 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:26.337203 (+ 484us) server_negotiation.cc:299] Negotiation successful
0808 15:53:26.337714 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":379,"threads_started":1}
I20260808 15:53:26.341746 22351 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55d5c4866e34119838be5d13f7da8c3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34627 } }
I20260808 15:53:26.342228 22351 raft_consensus.cc:374] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:26.342374 22351 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55d5c4866e34119838be5d13f7da8c3, State: Initialized, Role: FOLLOWER
I20260808 15:53:26.343083 22351 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55d5c4866e34119838be5d13f7da8c3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34627 } }
I20260808 15:53:26.343295 22351 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:26.343396 22351 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:26.343519 22351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:26.345723 22351 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55d5c4866e34119838be5d13f7da8c3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34627 } }
I20260808 15:53:26.346201 22351 leader_election.cc:302] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55d5c4866e34119838be5d13f7da8c3; no voters:
I20260808 15:53:26.346892 22351 leader_election.cc:288] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:26.347045 22358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:26.347766 22358 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [term 1 LEADER]: Becoming Leader. State: Replica: b55d5c4866e34119838be5d13f7da8c3, State: Running, Role: LEADER
I20260808 15:53:26.348433 22358 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55d5c4866e34119838be5d13f7da8c3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34627 } }
I20260808 15:53:26.349159 22351 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:26.351827 22359 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55d5c4866e34119838be5d13f7da8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55d5c4866e34119838be5d13f7da8c3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34627 } } }
I20260808 15:53:26.352018 22360 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55d5c4866e34119838be5d13f7da8c3. Latest consensus state: current_term: 1 leader_uuid: "b55d5c4866e34119838be5d13f7da8c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55d5c4866e34119838be5d13f7da8c3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34627 } } }
I20260808 15:53:26.352381 22360 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:26.352255 22359 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:26.353853 22373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:26.362190 22373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:26.370718 22373 catalog_manager.cc:1347] Generated new cluster ID: 603516ddfeba49de974efbf715970775
I20260808 15:53:26.370847 22373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:26.386713 22373 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:26.388067 22373 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:26.397562 22373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3: Generated new TSK 0
I20260808 15:53:26.398453 22373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:53:26.481711 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34627
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:26.838987 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:26.839299 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:26.839457 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:26.847697 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:26.847828 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:26.848027 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:26.858764 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:26.860100 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:26.861510 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:26.875301 22396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:26.875212 22397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:26.875250 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:26.876031 22390 server_base.cc:1034] running on GCE node
I20260808 15:53:26.876682 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:26.877686 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:26.878933 22390 hybrid_clock.cc:648] HybridClock initialized: now 1786204406878881 us; error 58 us; skew 500 ppm
I20260808 15:53:26.879283 22390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:26 dist-test-slave-twwt krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:26.882790 22390 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:53:26.884414 22390 webserver.cc:460] Webserver started at http://127.27.84.193:45087/ using document root <none> and password file <none>
I20260808 15:53:26.885142 22390 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:26.885290 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:26.885618 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:26.888666 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "3ad01952afea42018ce41fdabe5479f9"
format_stamp: "Formatted at 2026-08-08 15:53:26 on dist-test-slave-twwt"
server_key: "2d35f4e4f29355d65419bb0149b89899"
server_key_iv: "1b8832ee8069c82fa71af5473f1e627d"
server_key_version: "encryptionkey@0"
I20260808 15:53:26.889477 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "3ad01952afea42018ce41fdabe5479f9"
format_stamp: "Formatted at 2026-08-08 15:53:26 on dist-test-slave-twwt"
server_key: "2d35f4e4f29355d65419bb0149b89899"
server_key_iv: "1b8832ee8069c82fa71af5473f1e627d"
server_key_version: "encryptionkey@0"
I20260808 15:53:26.895116 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260808 15:53:26.898945 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:26.900527 22390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:26.900724 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "3ad01952afea42018ce41fdabe5479f9"
format_stamp: "Formatted at 2026-08-08 15:53:26 on dist-test-slave-twwt"
server_key: "2d35f4e4f29355d65419bb0149b89899"
server_key_iv: "1b8832ee8069c82fa71af5473f1e627d"
server_key_version: "encryptionkey@0"
I20260808 15:53:26.900956 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:26.923988 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:26.928712 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:26.929045 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:26.930140 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:26.931596 22390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:26.931701 22390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:26.931849 22390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:26.931917 22390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:26.967996 22390 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:40229
I20260808 15:53:26.968088 22568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:40229 every 8 connection(s)
I20260808 15:53:26.969370 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:26.976593 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 22390
I20260808 15:53:26.976821 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:53:26.977483 27987 external_mini_cluster.cc:1428] Setting key 071fdeced8b97ffc7e33912b6392b2b3
Aug 08 15:53:26 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:26.992902 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:26.972312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:49231 (local address 127.27.84.254:34627)
0808 15:53:26.972810 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:26.972815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:26.973859 (+ 1044us) server_negotiation.cc:407] Connection header received
0808 15:53:26.975854 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:26.975858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:26.975931 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:26.976036 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:26.977930 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:26.978920 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:26.980747 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:26.981497 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:26.981647 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:26.985429 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:26.985438 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:26.985445 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:26.985493 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:26.989224 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:26.990068 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:26.990077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:26.990083 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:26.990246 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:26.990750 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:26.990758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:26.990763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:26.991161 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:26.991327 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:26.992116 (+ 789us) server_negotiation.cc:299] Negotiation successful
0808 15:53:26.992400 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260808 15:53:26.993678 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:26.972936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34627 (local address 127.27.84.193:49231)
0808 15:53:26.973674 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:26.973725 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:26.975529 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:26.976339 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:26.976349 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:26.977060 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:26.977729 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:26.977751 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:26.979092 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:26.979098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:26.980584 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:26.980598 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:26.981749 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:26.981755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:26.981987 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:26.982779 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:26.982812 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:26.985181 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:26.989398 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:26.989406 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:26.989422 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:26.989935 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:26.990372 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:26.990376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:26.990378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:26.990582 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:26.991323 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:26.991329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:26.991691 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 15:53:26.991949 (+ 258us) client_negotiation.cc:240] Negotiation successful
0808 15:53:26.992401 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":285,"threads_started":1}
I20260808 15:53:26.995625 22569 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34627
I20260808 15:53:26.995980 22569 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:26.996742 22569 heartbeater.cc:511] Master 127.27.84.254:34627 requested a full tablet report, sending...
I20260808 15:53:26.999068 22299 ts_manager.cc:194] Registered new tserver with Master: 3ad01952afea42018ce41fdabe5479f9 (127.27.84.193:40229)
I20260808 15:53:27.000612 22299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:49231
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:53:27.055136 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34627
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:27.410226 22577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:27.410570 22577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:27.410751 22577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:27.418172 22577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:27.418334 22577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:27.418511 22577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:27.428445 22577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:27.429797 22577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:27.431227 22577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:27.444689 22590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:27.444783 22588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:27.444762 22587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:27.445564 22577 server_base.cc:1034] running on GCE node
I20260808 15:53:27.446208 22577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:27.447333 22577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:27.448603 22577 hybrid_clock.cc:648] HybridClock initialized: now 1786204407448552 us; error 41 us; skew 500 ppm
I20260808 15:53:27.448941 22577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:27 dist-test-slave-twwt krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204407, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:27.452265 22577 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:53:27.453953 22577 webserver.cc:460] Webserver started at http://127.27.84.194:41491/ using document root <none> and password file <none>
I20260808 15:53:27.454795 22577 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:27.454942 22577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:27.455271 22577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:27.458375 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18"
format_stamp: "Formatted at 2026-08-08 15:53:27 on dist-test-slave-twwt"
server_key: "12549661914b4215499422a04497685a"
server_key_iv: "9e9c04067ee7ee04ccaec1d8b7c20f82"
server_key_version: "encryptionkey@0"
I20260808 15:53:27.459200 22577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18"
format_stamp: "Formatted at 2026-08-08 15:53:27 on dist-test-slave-twwt"
server_key: "12549661914b4215499422a04497685a"
server_key_iv: "9e9c04067ee7ee04ccaec1d8b7c20f82"
server_key_version: "encryptionkey@0"
I20260808 15:53:27.464679 22577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:53:27.468218 22602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:27.469560 22577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:53:27.469735 22577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18"
format_stamp: "Formatted at 2026-08-08 15:53:27 on dist-test-slave-twwt"
server_key: "12549661914b4215499422a04497685a"
server_key_iv: "9e9c04067ee7ee04ccaec1d8b7c20f82"
server_key_version: "encryptionkey@0"
I20260808 15:53:27.469959 22577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:27.484784 22577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:27.489398 22577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:27.489712 22577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:27.490747 22577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:27.492134 22577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:27.492247 22577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:27.492390 22577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:27.492475 22577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:27.527242 22577 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:36575
I20260808 15:53:27.527288 22751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:36575 every 8 connection(s)
I20260808 15:53:27.528657 22577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:27.529151 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 22577
I20260808 15:53:27.529354 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:53:27.529830 27987 external_mini_cluster.cc:1428] Setting key 387ebc4bbb61683f63be088a6ebd4270
Aug 08 15:53:27 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204407, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:27.551633 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:27.531726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:57839 (local address 127.27.84.254:34627)
0808 15:53:27.532286 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:27.532292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:27.533514 (+ 1222us) server_negotiation.cc:407] Connection header received
0808 15:53:27.535613 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:27.535621 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:27.535718 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:27.535849 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:27.537544 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:27.538430 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:27.540107 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:27.540539 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:27.540679 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:27.544881 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:27.544890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:27.544898 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:27.544946 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:27.547981 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:27.548999 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:27.549002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:27.549005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:27.549148 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:27.549665 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:27.549668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:27.549670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:27.550033 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:27.550152 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:27.551161 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0808 15:53:27.551361 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":321,"threads_started":1}
I20260808 15:53:27.553278 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:27.532447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34627 (local address 127.27.84.194:57839)
0808 15:53:27.533317 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:27.533377 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:27.535340 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:27.536113 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:27.536123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:27.536694 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:27.537347 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:27.537367 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:27.538595 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:27.538601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:27.539939 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:27.539952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:27.540792 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:27.540801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:27.541051 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:27.541885 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:27.541910 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:27.544619 (+ 2709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:27.548135 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:27.548148 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:27.548169 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:27.548871 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:27.549258 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:27.549265 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:27.549270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:27.549533 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:27.550183 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:27.550195 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:27.550735 (+ 540us) client_negotiation.cc:769] Sending connection context
0808 15:53:27.551059 (+ 324us) client_negotiation.cc:240] Negotiation successful
0808 15:53:27.551658 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
I20260808 15:53:27.554857 22752 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34627
I20260808 15:53:27.555207 22752 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:27.556223 22752 heartbeater.cc:511] Master 127.27.84.254:34627 requested a full tablet report, sending...
I20260808 15:53:27.557922 22299 ts_manager.cc:194] Registered new tserver with Master: e18c99f3ac1c4bddbb3ccba2c40a4a18 (127.27.84.194:36575)
I20260808 15:53:27.558830 22299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:57839
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:53:27.610736 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34627
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:27.977600 22762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:27.977936 22762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:27.978094 22762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:27.985558 22762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:27.985705 22762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:27.985898 22762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:27.995887 22762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:34627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:27.997242 22762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:27.998589 22762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 15:53:28.003973 22569 heartbeater.cc:503] Master 127.27.84.254:34627 was elected leader, sending a full tablet report...
W20260808 15:53:28.012604 22771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:28.012660 22772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:28.013397 22774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:28.014201 22762 server_base.cc:1034] running on GCE node
I20260808 15:53:28.015030 22762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:28.016192 22762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:28.017448 22762 hybrid_clock.cc:648] HybridClock initialized: now 1786204408017395 us; error 37 us; skew 500 ppm
I20260808 15:53:28.017858 22762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:28 dist-test-slave-twwt krb5kdc[22232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204408, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:28.021541 22762 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:53:28.023558 22762 webserver.cc:460] Webserver started at http://127.27.84.195:46609/ using document root <none> and password file <none>
I20260808 15:53:28.024426 22762 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:28.024596 22762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:28.024988 22762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:28.028419 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "577b076f12d44701b5a9bf849daeca66"
format_stamp: "Formatted at 2026-08-08 15:53:28 on dist-test-slave-twwt"
server_key: "6f24afe62346569e21d1e95168e5bf64"
server_key_iv: "7bd48ee14086bb45409a143ddb5abd5b"
server_key_version: "encryptionkey@0"
I20260808 15:53:28.029327 22762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "577b076f12d44701b5a9bf849daeca66"
format_stamp: "Formatted at 2026-08-08 15:53:28 on dist-test-slave-twwt"
server_key: "6f24afe62346569e21d1e95168e5bf64"
server_key_iv: "7bd48ee14086bb45409a143ddb5abd5b"
server_key_version: "encryptionkey@0"
I20260808 15:53:28.035449 22762 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260808 15:53:28.039461 22784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:28.040997 22762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:28.041193 22762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "577b076f12d44701b5a9bf849daeca66"
format_stamp: "Formatted at 2026-08-08 15:53:28 on dist-test-slave-twwt"
server_key: "6f24afe62346569e21d1e95168e5bf64"
server_key_iv: "7bd48ee14086bb45409a143ddb5abd5b"
server_key_version: "encryptionkey@0"
I20260808 15:53:28.041419 22762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:28.062109 22762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:28.066578 22762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:28.066972 22762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:28.068030 22762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:28.069432 22762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:28.069514 22762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:28.069679 22762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:28.069730 22762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:28.104511 22762 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:33991
I20260808 15:53:28.104606 22957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:33991 every 8 connection(s)
I20260808 15:53:28.106033 22762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:28.115768 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 22762
I20260808 15:53:28.116011 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:53:28.116458 27987 external_mini_cluster.cc:1428] Setting key 450e85cc096c7cb40bfbc37b42cf954e
Aug 08 15:53:28 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204408, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:28.127075 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.109006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:49523 (local address 127.27.84.254:34627)
0808 15:53:28.109511 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.109516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.110652 (+ 1136us) server_negotiation.cc:407] Connection header received
0808 15:53:28.112571 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.112577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.112686 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.112876 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:28.114451 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.115352 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.117174 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.117689 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.117901 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.121532 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:28.121536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:28.121540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:28.121572 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:28.124294 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.125009 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.125012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.125015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.125130 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.125509 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.125513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.125514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.125764 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:28.125921 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.126555 (+ 634us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.126772 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":240,"threads_started":1}
I20260808 15:53:28.128077 22970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.109657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:34627 (local address 127.27.84.195:49523)
0808 15:53:28.110458 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:28.110498 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:28.112282 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:28.113090 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:28.113099 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:28.113671 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:28.114249 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.114266 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.115530 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.115534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.116992 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.117010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.117961 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.117970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.118230 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.118987 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:28.119018 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:28.121348 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:28.124440 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:28.124447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:28.124459 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:28.124904 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:28.125221 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:28.125224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:28.125226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:28.125419 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:28.125900 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:28.125906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:28.126212 (+ 306us) client_negotiation.cc:769] Sending connection context
0808 15:53:28.126459 (+ 247us) client_negotiation.cc:240] Negotiation successful
0808 15:53:28.126913 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":292,"threads_started":1}
I20260808 15:53:28.129495 22958 heartbeater.cc:347] Connected to a master server at 127.27.84.254:34627
I20260808 15:53:28.129813 22958 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:28.130658 22958 heartbeater.cc:511] Master 127.27.84.254:34627 requested a full tablet report, sending...
I20260808 15:53:28.132171 22299 ts_manager.cc:194] Registered new tserver with Master: 577b076f12d44701b5a9bf849daeca66 (127.27.84.195:33991)
I20260808 15:53:28.133008 22299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:49523
I20260808 15:53:28.144407 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:53:28.163419 22964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.149444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.27.84.254:34627)
0808 15:53:28.149621 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.149625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.150146 (+ 521us) server_negotiation.cc:407] Connection header received
0808 15:53:28.150268 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.150272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.150344 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.150480 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:28.152074 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.152836 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.154234 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.154707 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.154857 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.157913 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:28.157918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:28.157922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:28.157957 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:28.160528 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.161252 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.161258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.161262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.161401 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.161787 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.161792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.161795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.162140 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:28.162415 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.162847 (+ 432us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.163098 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:53:28.168098 22299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:53:28.175712 22299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:53:28.214272 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.188564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:33991 (local address 127.0.0.1:50834)
0808 15:53:28.201830 (+ 13266us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:28.201876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:28.202056 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:28.205839 (+ 3783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:28.205852 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:28.205897 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:28.206477 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.206493 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.208776 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.208783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.211275 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.211295 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.213190 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.213196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.213391 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.213447 (+ 56us) client_negotiation.cc:769] Sending connection context
0808 15:53:28.213593 (+ 146us) client_negotiation.cc:240] Negotiation successful
0808 15:53:28.213791 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12987,"thread_start_us":15142,"threads_started":1}
I20260808 15:53:28.215430 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.188853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.27.84.195:33991)
0808 15:53:28.204485 (+ 15632us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.204499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.204546 (+ 47us) server_negotiation.cc:407] Connection header received
0808 15:53:28.204666 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.204674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.205041 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.205416 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:28.206744 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.208610 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.211473 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.212996 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.213130 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.213390 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.214904 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.215083 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15398,"thread_start_us":15231,"threads_started":1}
I20260808 15:53:28.215456 22992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.189313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:36575 (local address 127.0.0.1:43048)
0808 15:53:28.206367 (+ 17054us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:28.206389 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:28.206532 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:28.207566 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:28.207571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:28.207597 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:28.208069 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.208084 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.210173 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.210180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.212443 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.212459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.214672 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.214681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.214892 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.214932 (+ 40us) client_negotiation.cc:769] Sending connection context
0808 15:53:28.215021 (+ 89us) client_negotiation.cc:240] Negotiation successful
0808 15:53:28.215136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16915,"mutex_wait_us":24,"thread_start_us":16717,"threads_started":1}
I20260808 15:53:28.215533 22983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.188754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40229 (local address 127.0.0.1:40816)
0808 15:53:28.203473 (+ 14719us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:28.203494 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:28.203616 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:28.208584 (+ 4968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:28.208589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:28.208616 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:28.209131 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.209142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.211040 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.211045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.212685 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.212718 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.214593 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.214600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.214814 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.214853 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:53:28.214925 (+ 72us) client_negotiation.cc:240] Negotiation successful
0808 15:53:28.215052 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14583,"mutex_wait_us":6,"thread_start_us":12637,"threads_started":1}
I20260808 15:53:28.216202 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.189942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.27.84.194:36575)
0808 15:53:28.206815 (+ 16873us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.206829 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.206866 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:53:28.206992 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.207003 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.207322 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.207535 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:28.208308 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.210013 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.212635 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.214103 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.214253 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.214480 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.215789 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.215969 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16609,"thread_start_us":16919,"threads_started":1}
I20260808 15:53:28.216661 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.189299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.27.84.193:40229)
0808 15:53:28.208008 (+ 18709us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.208017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.208046 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:53:28.208135 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.208143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.208359 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.209299 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:28.209505 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.210834 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.212880 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.214338 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.215871 (+ 1533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.216090 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.216155 (+ 65us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.216332 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18157,"thread_start_us":17976,"threads_started":1}
I20260808 15:53:28.219652 22666 tablet_service.cc:1467] Processing CreateTablet for tablet 4c47e568c12b47239bed385a18ac5b8c (DEFAULT_TABLE table=test-table [id=a143009608564476bcd22cd71c089416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:28.219739 22851 tablet_service.cc:1467] Processing CreateTablet for tablet 4c47e568c12b47239bed385a18ac5b8c (DEFAULT_TABLE table=test-table [id=a143009608564476bcd22cd71c089416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:28.220024 22474 tablet_service.cc:1467] Processing CreateTablet for tablet 4c47e568c12b47239bed385a18ac5b8c (DEFAULT_TABLE table=test-table [id=a143009608564476bcd22cd71c089416]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:28.221192 22666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c47e568c12b47239bed385a18ac5b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:28.221282 22851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c47e568c12b47239bed385a18ac5b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:28.221262 22474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c47e568c12b47239bed385a18ac5b8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:28.233141 22995 tablet_bootstrap.cc:492] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Bootstrap starting.
I20260808 15:53:28.236420 22998 tablet_bootstrap.cc:492] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Bootstrap starting.
I20260808 15:53:28.237115 22995 tablet_bootstrap.cc:654] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:28.237254 22997 tablet_bootstrap.cc:492] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Bootstrap starting.
I20260808 15:53:28.238669 22995 log.cc:824] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:28.240880 22998 tablet_bootstrap.cc:654] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:28.240978 22997 tablet_bootstrap.cc:654] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:28.242256 22998 log.cc:824] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:28.242349 22997 log.cc:824] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:28.244776 22995 tablet_bootstrap.cc:492] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: No bootstrap required, opened a new log
I20260808 15:53:28.245081 22995 ts_tablet_manager.cc:1397] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260808 15:53:28.245715 22998 tablet_bootstrap.cc:492] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: No bootstrap required, opened a new log
I20260808 15:53:28.246029 22998 ts_tablet_manager.cc:1397] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260808 15:53:28.248484 22997 tablet_bootstrap.cc:492] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: No bootstrap required, opened a new log
I20260808 15:53:28.248751 22997 ts_tablet_manager.cc:1397] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260808 15:53:28.253703 22998 raft_consensus.cc:348] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.254292 22998 raft_consensus.cc:374] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:28.254482 22998 raft_consensus.cc:729] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 577b076f12d44701b5a9bf849daeca66, State: Initialized, Role: FOLLOWER
I20260808 15:53:28.255324 22998 consensus_queue.cc:260] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.256531 22958 heartbeater.cc:503] Master 127.27.84.254:34627 was elected leader, sending a full tablet report...
I20260808 15:53:28.257066 22998 ts_tablet_manager.cc:1428] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Time spent starting tablet: real 0.011s user 0.001s sys 0.010s
I20260808 15:53:28.256198 22995 raft_consensus.cc:348] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.257606 22995 raft_consensus.cc:374] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:28.257725 22995 raft_consensus.cc:729] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18c99f3ac1c4bddbb3ccba2c40a4a18, State: Initialized, Role: FOLLOWER
I20260808 15:53:28.258440 22995 consensus_queue.cc:260] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.259613 22752 heartbeater.cc:503] Master 127.27.84.254:34627 was elected leader, sending a full tablet report...
I20260808 15:53:28.259835 22997 raft_consensus.cc:348] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.260243 22995 ts_tablet_manager.cc:1428] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260808 15:53:28.260252 22997 raft_consensus.cc:374] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:28.260360 22997 raft_consensus.cc:729] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ad01952afea42018ce41fdabe5479f9, State: Initialized, Role: FOLLOWER
I20260808 15:53:28.261014 22997 consensus_queue.cc:260] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.262802 22997 ts_tablet_manager.cc:1428] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260808 15:53:28.281723 22753 tablet.cc:2369] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:28.282389 23006 raft_consensus.cc:482] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:28.282739 23006 raft_consensus.cc:504] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.284499 23006 leader_election.cc:288] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 577b076f12d44701b5a9bf849daeca66 (127.27.84.195:33991), 3ad01952afea42018ce41fdabe5479f9 (127.27.84.193:40229)
I20260808 15:53:28.292394 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.285103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:38959 (local address 127.27.84.193:40229)
0808 15:53:28.285341 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.285350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.286181 (+ 831us) server_negotiation.cc:407] Connection header received
0808 15:53:28.286370 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.286374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.286435 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.286506 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:28.287235 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.287985 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.290046 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.291356 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.291524 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.291619 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.291992 (+ 373us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.292130 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:53:28.292423 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.285003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:40637 (local address 127.27.84.195:33991)
0808 15:53:28.285223 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.285229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.286065 (+ 836us) server_negotiation.cc:407] Connection header received
0808 15:53:28.286384 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.286389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.286452 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.286558 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:28.287296 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.288031 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.290217 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.291492 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.291644 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.291716 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.292028 (+ 312us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.292155 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:53:28.292588 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.285002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:33991 (local address 127.27.84.194:40637)
0808 15:53:28.285930 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:28.285953 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:28.286097 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:28.286596 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:28.286600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:28.286647 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:28.287130 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.287141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.288194 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.288197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.290006 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.290022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.291699 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.291704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.291855 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.291903 (+ 48us) client_negotiation.cc:769] Sending connection context
0808 15:53:28.291984 (+ 81us) client_negotiation.cc:240] Negotiation successful
0808 15:53:28.292090 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":65,"thread_start_us":490,"threads_started":1}
I20260808 15:53:28.292999 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.285182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:40229 (local address 127.27.84.194:38959)
0808 15:53:28.286044 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:28.286063 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:28.286219 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:28.286556 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:28.286561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:28.286584 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:28.287063 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.287074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.288119 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.288125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.289830 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:28.289843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.291584 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.291591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:28.291792 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.291830 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:53:28.291934 (+ 104us) client_negotiation.cc:240] Negotiation successful
0808 15:53:28.292091 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":69,"thread_start_us":481,"threads_started":1}
I20260808 15:53:28.293749 22885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c47e568c12b47239bed385a18ac5b8c" candidate_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577b076f12d44701b5a9bf849daeca66" is_pre_election: true
I20260808 15:53:28.294029 22498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c47e568c12b47239bed385a18ac5b8c" candidate_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad01952afea42018ce41fdabe5479f9" is_pre_election: true
I20260808 15:53:28.294282 22885 raft_consensus.cc:2454] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18c99f3ac1c4bddbb3ccba2c40a4a18 in term 0.
I20260808 15:53:28.294579 22498 raft_consensus.cc:2454] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18c99f3ac1c4bddbb3ccba2c40a4a18 in term 0.
I20260808 15:53:28.295032 22606 leader_election.cc:302] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577b076f12d44701b5a9bf849daeca66, e18c99f3ac1c4bddbb3ccba2c40a4a18; no voters:
I20260808 15:53:28.295403 23006 raft_consensus.cc:2781] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:53:28.295562 23006 raft_consensus.cc:482] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:53:28.295667 23006 raft_consensus.cc:3037] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:28.297907 23006 raft_consensus.cc:504] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.298558 23006 leader_election.cc:288] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [CANDIDATE]: Term 1 election: Requested vote from peers 577b076f12d44701b5a9bf849daeca66 (127.27.84.195:33991), 3ad01952afea42018ce41fdabe5479f9 (127.27.84.193:40229)
I20260808 15:53:28.299155 22885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c47e568c12b47239bed385a18ac5b8c" candidate_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "577b076f12d44701b5a9bf849daeca66"
I20260808 15:53:28.299155 22498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c47e568c12b47239bed385a18ac5b8c" candidate_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ad01952afea42018ce41fdabe5479f9"
I20260808 15:53:28.299389 22885 raft_consensus.cc:3037] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:28.299396 22498 raft_consensus.cc:3037] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:28.301743 22885 raft_consensus.cc:2454] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18c99f3ac1c4bddbb3ccba2c40a4a18 in term 1.
I20260808 15:53:28.301748 22498 raft_consensus.cc:2454] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18c99f3ac1c4bddbb3ccba2c40a4a18 in term 1.
I20260808 15:53:28.302285 22606 leader_election.cc:302] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 577b076f12d44701b5a9bf849daeca66, e18c99f3ac1c4bddbb3ccba2c40a4a18; no voters:
I20260808 15:53:28.302659 23006 raft_consensus.cc:2781] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:28.303160 23006 raft_consensus.cc:686] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 1 LEADER]: Becoming Leader. State: Replica: e18c99f3ac1c4bddbb3ccba2c40a4a18, State: Running, Role: LEADER
I20260808 15:53:28.303759 23006 consensus_queue.cc:237] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } }
I20260808 15:53:28.309311 22299 catalog_manager.cc:5495] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 reported cstate change: term changed from 0 to 1, leader changed from <none> to e18c99f3ac1c4bddbb3ccba2c40a4a18 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e18c99f3ac1c4bddbb3ccba2c40a4a18" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 36575 } health_report { overall_health: HEALTHY } } }
I20260808 15:53:28.327524 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.321847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.27.84.194:36575)
0808 15:53:28.322085 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.322093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.322181 (+ 88us) server_negotiation.cc:407] Connection header received
0808 15:53:28.322386 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.322390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.322467 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.322587 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:53:28.323032 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.323742 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.325183 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.325610 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.325712 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.326258 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:53:28.326891 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:53:28.327018 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.327147 (+ 129us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.327220 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260808 15:53:28.335923 22885 raft_consensus.cc:1261] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 1 FOLLOWER]: Refusing update from remote peer e18c99f3ac1c4bddbb3ccba2c40a4a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:28.335924 22498 raft_consensus.cc:1261] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 1 FOLLOWER]: Refusing update from remote peer e18c99f3ac1c4bddbb3ccba2c40a4a18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:28.336963 23010 consensus_queue.cc:1035] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ad01952afea42018ce41fdabe5479f9" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:28.337338 23006 consensus_queue.cc:1035] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "577b076f12d44701b5a9bf849daeca66" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:28.349895 23013 mvcc.cc:205] Tried to move back new op lower bound from 7316293256533352448 to 7316293256416722944. Current Snapshot: MvccSnapshot[applied={T|T < 7316293256533352448}]
I20260808 15:53:28.351652 23016 mvcc.cc:205] Tried to move back new op lower bound from 7316293256533352448 to 7316293256416722944. Current Snapshot: MvccSnapshot[applied={T|T < 7316293256533352448}]
I20260808 15:53:28.367118 22298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:28.367430 22298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:28.371382 22298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b55d5c4866e34119838be5d13f7da8c3: Sending DeleteTablet for 3 replicas of tablet 4c47e568c12b47239bed385a18ac5b8c
I20260808 15:53:28.372524 22851 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c47e568c12b47239bed385a18ac5b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:28 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:50834
I20260808 15:53:28.372641 22474 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c47e568c12b47239bed385a18ac5b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:28 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:40816
I20260808 15:53:28.372684 22666 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c47e568c12b47239bed385a18ac5b8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:28 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:43048
I20260808 15:53:28.376133 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 22390
I20260808 15:53:28.389161 23032 tablet_replica.cc:330] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: stopping tablet replica
I20260808 15:53:28.389901 23032 raft_consensus.cc:2229] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:28.390656 23032 raft_consensus.cc:2258] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:28.393136 23032 ts_tablet_manager.cc:1905] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:28.400908 23032 ts_tablet_manager.cc:1918] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:28.401232 23032 log.cc:1196] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/4c47e568c12b47239bed385a18ac5b8c
I20260808 15:53:28.401984 23032 ts_tablet_manager.cc:1939] T 4c47e568c12b47239bed385a18ac5b8c P 3ad01952afea42018ce41fdabe5479f9: Deleting consensus metadata
I20260808 15:53:28.404423 22282 catalog_manager.cc:4841] TS 3ad01952afea42018ce41fdabe5479f9 (127.27.84.193:40229): tablet 4c47e568c12b47239bed385a18ac5b8c (table test-table [id=a143009608564476bcd22cd71c089416]) successfully deleted
I20260808 15:53:28.408964 23028 tablet_replica.cc:330] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: stopping tablet replica
I20260808 15:53:28.409771 23029 tablet_replica.cc:330] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: stopping tablet replica
I20260808 15:53:28.410383 23029 raft_consensus.cc:2229] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:53:28.410377 23028 raft_consensus.cc:2229] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:28.411000 23028 raft_consensus.cc:2258] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:28.411234 23029 raft_consensus.cc:2258] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 08 15:53:28 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:53:28.413515 23028 ts_tablet_manager.cc:1905] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:28.413522 23029 ts_tablet_manager.cc:1905] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:28.421306 23029 ts_tablet_manager.cc:1918] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:28.421501 23028 ts_tablet_manager.cc:1918] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:28.421597 23029 log.cc:1196] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/4c47e568c12b47239bed385a18ac5b8c
I20260808 15:53:28.421695 23028 log.cc:1196] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/4c47e568c12b47239bed385a18ac5b8c
I20260808 15:53:28.422314 23029 ts_tablet_manager.cc:1939] T 4c47e568c12b47239bed385a18ac5b8c P e18c99f3ac1c4bddbb3ccba2c40a4a18: Deleting consensus metadata
I20260808 15:53:28.422308 23028 ts_tablet_manager.cc:1939] T 4c47e568c12b47239bed385a18ac5b8c P 577b076f12d44701b5a9bf849daeca66: Deleting consensus metadata
I20260808 15:53:28.423043 22991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.378048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40826 (local address 127.27.84.193:40229)
0808 15:53:28.378291 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.378311 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.399585 (+ 21274us) server_negotiation.cc:407] Connection header received
0808 15:53:28.400236 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.400244 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.400370 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.400468 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:28.403381 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.404982 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.407821 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.408501 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.408644 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.414758 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:28.414777 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:28.414797 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:28.414850 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:28.419040 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.420023 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.420034 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.420037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.420209 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.420747 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.420753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.420757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.421311 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:28.421661 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.422355 (+ 694us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.422628 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:53:28.424190 22283 catalog_manager.cc:4841] TS 577b076f12d44701b5a9bf849daeca66 (127.27.84.195:33991): tablet 4c47e568c12b47239bed385a18ac5b8c (table test-table [id=a143009608564476bcd22cd71c089416]) successfully deleted
I20260808 15:53:28.424599 22281 catalog_manager.cc:4841] TS e18c99f3ac1c4bddbb3ccba2c40a4a18 (127.27.84.194:36575): tablet 4c47e568c12b47239bed385a18ac5b8c (table test-table [id=a143009608564476bcd22cd71c089416]) successfully deleted
I20260808 15:53:28.424906 22563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:28.517495 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 22390
I20260808 15:53:28.553234 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 22577
Aug 08 15:53:28 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:53:28.571699 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.554073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.27.84.194:36575)
0808 15:53:28.554302 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.554308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.554331 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:53:28.554499 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.554505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.554674 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.554811 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:28.556397 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.557612 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.559146 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.559823 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.559958 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.564502 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:28.564518 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:28.564546 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:28.564599 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:28.568688 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.569404 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.569418 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.569422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.569585 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.569960 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.569965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.569969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.570462 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:28.570829 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.571043 (+ 214us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.571293 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:53:28.572566 22747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:28.689217 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 22577
I20260808 15:53:28.716789 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 22762
Aug 08 15:53:28 dist-test-slave-twwt krb5kdc[22232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:53:28.736924 22990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:28.717559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.27.84.195:33991)
0808 15:53:28.717737 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:28.717742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:28.717764 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:53:28.717974 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:28.717978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:28.718044 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:28.718140 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:28.719657 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.720424 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.721739 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:28.722159 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:28.722256 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:28.730992 (+ 8736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:28.731014 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:28.731031 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:28.731075 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:28.734111 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.734996 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.735005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.735007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.735173 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:28.735520 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:28.735523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:28.735525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:28.735923 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:28.736156 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:28.736463 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 15:53:28.736647 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:53:28.737622 22950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:28.831980 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 22762
I20260808 15:53:28.858254 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 22254
I20260808 15:53:28.859591 22343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:28.972676 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 22254
2026-08-08T15:53:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4926 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-08T15:53:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:29Z Disabled control of system clock
I20260808 15:53:29.082177 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42307
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:53:29.440929 23045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:29.441227 23045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:29.441311 23045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:29.448447 23045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:29.448566 23045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:29.448606 23045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:29.448657 23045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:29.448702 23045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:29.457983 23045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42307
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:29.459390 23045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:29.461187 23045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:29.473254 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:29.473213 23056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:29.473286 23054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:29.474469 23045 server_base.cc:1034] running on GCE node
I20260808 15:53:29.475328 23045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:29.476769 23045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:29.478016 23045 hybrid_clock.cc:648] HybridClock initialized: now 1786204409477973 us; error 25 us; skew 500 ppm
I20260808 15:53:29.478444 23045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:29.481014 23045 webserver.cc:460] Webserver started at http://127.27.84.254:42575/ using document root <none> and password file <none>
I20260808 15:53:29.481776 23045 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:29.481905 23045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:29.482270 23045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:29.485332 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "45caf07029ad46e6a70a36c63009602d"
format_stamp: "Formatted at 2026-08-08 15:53:29 on dist-test-slave-twwt"
server_key: "1c298c4ff91c43a2b4802a0d2c3748c6"
server_key_iv: "345cae96d270887947bbde787884aac4"
server_key_version: "encryptionkey@0"
I20260808 15:53:29.486191 23045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "45caf07029ad46e6a70a36c63009602d"
format_stamp: "Formatted at 2026-08-08 15:53:29 on dist-test-slave-twwt"
server_key: "1c298c4ff91c43a2b4802a0d2c3748c6"
server_key_iv: "345cae96d270887947bbde787884aac4"
server_key_version: "encryptionkey@0"
I20260808 15:53:29.491554 23045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260808 15:53:29.495105 23065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:29.496503 23045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:53:29.496703 23045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "45caf07029ad46e6a70a36c63009602d"
format_stamp: "Formatted at 2026-08-08 15:53:29 on dist-test-slave-twwt"
server_key: "1c298c4ff91c43a2b4802a0d2c3748c6"
server_key_iv: "345cae96d270887947bbde787884aac4"
server_key_version: "encryptionkey@0"
I20260808 15:53:29.496937 23045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:29.525127 23045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:29.526109 23045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:29.526453 23045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:29.546463 23045 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:42307
I20260808 15:53:29.546481 23143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:42307 every 8 connection(s)
I20260808 15:53:29.548015 23045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:29.552107 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:29.556349 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 23045
I20260808 15:53:29.556545 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:29.557135 27987 external_mini_cluster.cc:1428] Setting key 3603a665d33669889eaa0027061d62ec
I20260808 15:53:29.565794 23145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d: Bootstrap starting.
I20260808 15:53:29.569180 23145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:29.570449 23145 log.cc:824] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:29.570928 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:29.559264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.27.84.254:42307)
0808 15:53:29.560259 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:29.560271 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:29.560320 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:53:29.561945 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:29.561969 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:29.562479 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:29.562938 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:29.563547 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:29.564869 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:29.566309 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:29.567075 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:29.567297 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:29.568501 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:29.568509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:29.568523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:29.569053 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:29.569163 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:29.569188 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:29.569328 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:29.569565 (+ 237us) server_negotiation.cc:299] Negotiation successful
0808 15:53:29.569887 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":363,"threads_started":1}
I20260808 15:53:29.573611 23145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d: No bootstrap required, opened a new log
I20260808 15:53:29.581651 23145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45caf07029ad46e6a70a36c63009602d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42307 } }
I20260808 15:53:29.582079 23145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:29.582190 23145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45caf07029ad46e6a70a36c63009602d, State: Initialized, Role: FOLLOWER
I20260808 15:53:29.582991 23145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45caf07029ad46e6a70a36c63009602d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42307 } }
I20260808 15:53:29.583302 23145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:29.583472 23145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:29.583676 23145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:29.586139 23145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45caf07029ad46e6a70a36c63009602d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42307 } }
I20260808 15:53:29.586720 23145 leader_election.cc:302] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45caf07029ad46e6a70a36c63009602d; no voters:
I20260808 15:53:29.587379 23145 leader_election.cc:288] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:29.587565 23150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:29.588353 23150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [term 1 LEADER]: Becoming Leader. State: Replica: 45caf07029ad46e6a70a36c63009602d, State: Running, Role: LEADER
I20260808 15:53:29.588919 23150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45caf07029ad46e6a70a36c63009602d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42307 } }
I20260808 15:53:29.589749 23145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:29.592152 23153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45caf07029ad46e6a70a36c63009602d. Latest consensus state: current_term: 1 leader_uuid: "45caf07029ad46e6a70a36c63009602d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45caf07029ad46e6a70a36c63009602d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42307 } } }
I20260808 15:53:29.592659 23153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:29.593617 23151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45caf07029ad46e6a70a36c63009602d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45caf07029ad46e6a70a36c63009602d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42307 } } }
I20260808 15:53:29.594005 23151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:29.597579 23159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:29.602567 23159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:29.611877 23159 catalog_manager.cc:1347] Generated new cluster ID: e291ff79ebab4ead9199f1a1282b7c19
I20260808 15:53:29.611997 23159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:29.626914 23159 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:29.628799 23159 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:29.637677 23159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d: Generated new TSK 0
I20260808 15:53:29.638868 23159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:29.654633 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42307
--builtin_ntp_servers=127.27.84.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:53:30.026527 23181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:30.026998 23181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:30.027174 23181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:30.034847 23181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:30.034997 23181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:30.035183 23181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:30.044843 23181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:30.046291 23181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:30.048314 23181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:30.062271 23187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:30.062285 23188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:30.062633 23190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:30.063153 23181 server_base.cc:1034] running on GCE node
I20260808 15:53:30.063817 23181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:30.064845 23181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:30.066108 23181 hybrid_clock.cc:648] HybridClock initialized: now 1786204410066048 us; error 46 us; skew 500 ppm
I20260808 15:53:30.066468 23181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:30.069082 23181 webserver.cc:460] Webserver started at http://127.27.84.193:39739/ using document root <none> and password file <none>
I20260808 15:53:30.069860 23181 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:30.069998 23181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:30.070442 23181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:30.073598 23181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "69ad09b17cb24c3594b82f72cb266421"
format_stamp: "Formatted at 2026-08-08 15:53:30 on dist-test-slave-twwt"
server_key: "503a050edd6631a779e2992bd805ae9f"
server_key_iv: "5361ad6758afa14c6fea7e78f50e6d24"
server_key_version: "encryptionkey@0"
I20260808 15:53:30.074434 23181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "69ad09b17cb24c3594b82f72cb266421"
format_stamp: "Formatted at 2026-08-08 15:53:30 on dist-test-slave-twwt"
server_key: "503a050edd6631a779e2992bd805ae9f"
server_key_iv: "5361ad6758afa14c6fea7e78f50e6d24"
server_key_version: "encryptionkey@0"
I20260808 15:53:30.079748 23181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260808 15:53:30.084079 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:30.085474 23181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:30.085638 23181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "69ad09b17cb24c3594b82f72cb266421"
format_stamp: "Formatted at 2026-08-08 15:53:30 on dist-test-slave-twwt"
server_key: "503a050edd6631a779e2992bd805ae9f"
server_key_iv: "5361ad6758afa14c6fea7e78f50e6d24"
server_key_version: "encryptionkey@0"
I20260808 15:53:30.085870 23181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:30.105834 23181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:30.106752 23181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:30.107079 23181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:30.108129 23181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:30.109699 23181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:30.109844 23181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:30.110018 23181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:30.110136 23181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:30.147639 23181 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:39651
I20260808 15:53:30.147735 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:39651 every 8 connection(s)
I20260808 15:53:30.149034 23181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:30.150902 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 23181
I20260808 15:53:30.151085 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:53:30.151598 27987 external_mini_cluster.cc:1428] Setting key 7a102f24f74c1b8d53c8b301f22f84b5
I20260808 15:53:30.154901 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42307
--builtin_ntp_servers=127.27.84.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:53:30.162768 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:30.152017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:45225 (local address 127.27.84.254:42307)
0808 15:53:30.152536 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:30.152543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:30.153972 (+ 1429us) server_negotiation.cc:407] Connection header received
0808 15:53:30.156038 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:30.156044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:30.156136 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:30.156258 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:30.157376 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.158148 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.159840 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.160246 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.160409 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:30.161841 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:30.161847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:30.161856 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:30.161875 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:30.161917 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:30.161929 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:30.162031 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:30.162261 (+ 230us) server_negotiation.cc:299] Negotiation successful
0808 15:53:30.162371 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":260,"threads_started":1}
I20260808 15:53:30.163780 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:30.152662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42307 (local address 127.27.84.193:45225)
0808 15:53:30.153800 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:30.153864 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:30.155749 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:30.156506 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:30.156514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:30.156606 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:30.157185 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:30.157201 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.158316 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.158321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:30.159676 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:30.159690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.160528 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.160536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:30.160766 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:30.161505 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:30.161508 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:30.161549 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:30.161633 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:30.162033 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:30.162048 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:30.162058 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:53:30.162249 (+ 191us) client_negotiation.cc:240] Negotiation successful
0808 15:53:30.162679 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":371,"threads_started":1}
I20260808 15:53:30.165493 23361 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42307
I20260808 15:53:30.165830 23361 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:30.166738 23361 heartbeater.cc:511] Master 127.27.84.254:42307 requested a full tablet report, sending...
I20260808 15:53:30.169165 23092 ts_manager.cc:194] Registered new tserver with Master: 69ad09b17cb24c3594b82f72cb266421 (127.27.84.193:39651)
I20260808 15:53:30.171496 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.193:45225
W20260808 15:53:30.506333 23366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:30.506736 23366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:30.506920 23366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:30.514421 23366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:30.514519 23366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:30.514732 23366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:30.523904 23366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:30.525310 23366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:30.527194 23366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:30.540700 23375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:30.540726 23376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:30.540779 23379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:30.542130 23366 server_base.cc:1034] running on GCE node
I20260808 15:53:30.542930 23366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:30.543969 23366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:30.545230 23366 hybrid_clock.cc:648] HybridClock initialized: now 1786204410545185 us; error 31 us; skew 500 ppm
I20260808 15:53:30.545710 23366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:30.548420 23366 webserver.cc:460] Webserver started at http://127.27.84.194:45117/ using document root <none> and password file <none>
I20260808 15:53:30.549187 23366 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:30.549345 23366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:30.549702 23366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:30.553028 23366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "bd3081f3f14f419f88c9ed32a6b3d617"
format_stamp: "Formatted at 2026-08-08 15:53:30 on dist-test-slave-twwt"
server_key: "565cba92e41a844a9f76b13a84a6ef52"
server_key_iv: "b222943d6d15333ef55b1f940469f0be"
server_key_version: "encryptionkey@0"
I20260808 15:53:30.553922 23366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "bd3081f3f14f419f88c9ed32a6b3d617"
format_stamp: "Formatted at 2026-08-08 15:53:30 on dist-test-slave-twwt"
server_key: "565cba92e41a844a9f76b13a84a6ef52"
server_key_iv: "b222943d6d15333ef55b1f940469f0be"
server_key_version: "encryptionkey@0"
I20260808 15:53:30.559746 23366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260808 15:53:30.563592 23387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:30.565061 23366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:30.565394 23366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "bd3081f3f14f419f88c9ed32a6b3d617"
format_stamp: "Formatted at 2026-08-08 15:53:30 on dist-test-slave-twwt"
server_key: "565cba92e41a844a9f76b13a84a6ef52"
server_key_iv: "b222943d6d15333ef55b1f940469f0be"
server_key_version: "encryptionkey@0"
I20260808 15:53:30.565672 23366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:30.580572 23366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:30.581471 23366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:30.581835 23366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:30.582945 23366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:30.584645 23366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:30.584803 23366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:30.584945 23366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:30.585047 23366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:30.624562 23366 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:35333
I20260808 15:53:30.624614 23564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:35333 every 8 connection(s)
I20260808 15:53:30.626119 23366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:30.631733 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 23366
I20260808 15:53:30.631930 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:53:30.632510 27987 external_mini_cluster.cc:1428] Setting key 7c7690b8ce30ae60b55c9b10ae8cc578
I20260808 15:53:30.635831 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42307
--builtin_ntp_servers=127.27.84.212:41515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:53:30.641868 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:30.629080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36913 (local address 127.27.84.254:42307)
0808 15:53:30.629276 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:30.629281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:30.630813 (+ 1532us) server_negotiation.cc:407] Connection header received
0808 15:53:30.633367 (+ 2554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:30.633373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:30.633466 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:30.633588 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:30.635021 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.635829 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.638465 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.639050 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.639172 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:30.640990 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:30.640996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:30.641000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:30.641022 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:30.641059 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:30.641073 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:30.641165 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:30.641472 (+ 307us) server_negotiation.cc:299] Negotiation successful
0808 15:53:30.641569 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260808 15:53:30.643432 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:30.629613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42307 (local address 127.27.84.194:36913)
0808 15:53:30.630646 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:30.630699 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:30.633027 (+ 2328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:30.633860 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:30.633873 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:30.633989 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:30.634797 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:30.634819 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.636014 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.636021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:30.638277 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:30.638296 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:30.639306 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:30.639315 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:30.639682 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:30.640567 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:30.640573 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:30.640632 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:30.640747 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:30.641190 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:30.641204 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:30.641215 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:53:30.641415 (+ 200us) client_negotiation.cc:240] Negotiation successful
0808 15:53:30.641892 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":303,"threads_started":1}
I20260808 15:53:30.645135 23566 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42307
I20260808 15:53:30.645478 23566 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:30.646454 23566 heartbeater.cc:511] Master 127.27.84.254:42307 requested a full tablet report, sending...
I20260808 15:53:30.648223 23092 ts_manager.cc:194] Registered new tserver with Master: bd3081f3f14f419f88c9ed32a6b3d617 (127.27.84.194:35333)
I20260808 15:53:30.649138 23092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.194:36913
W20260808 15:53:31.014503 23569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:31.014953 23569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:31.015115 23569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:31.022919 23569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:31.023032 23569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:31.023214 23569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:31.032954 23569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:31.034409 23569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:31.036424 23569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:31.051036 23576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:31.051091 23580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:31.051079 23577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:31.051805 23569 server_base.cc:1034] running on GCE node
I20260808 15:53:31.052446 23569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:31.053498 23569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:31.054720 23569 hybrid_clock.cc:648] HybridClock initialized: now 1786204411054628 us; error 84 us; skew 500 ppm
I20260808 15:53:31.055054 23569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:31.057426 23569 webserver.cc:460] Webserver started at http://127.27.84.195:35789/ using document root <none> and password file <none>
I20260808 15:53:31.058305 23569 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:31.058406 23569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:31.058745 23569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:31.061879 23569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "d0993dafdf0c4c2f81ed231df30fd8ca"
format_stamp: "Formatted at 2026-08-08 15:53:31 on dist-test-slave-twwt"
server_key: "1116214dec7f9671b903abe9697667c0"
server_key_iv: "1c405f2be996063b14fe28bc4e154b67"
server_key_version: "encryptionkey@0"
I20260808 15:53:31.062783 23569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "d0993dafdf0c4c2f81ed231df30fd8ca"
format_stamp: "Formatted at 2026-08-08 15:53:31 on dist-test-slave-twwt"
server_key: "1116214dec7f9671b903abe9697667c0"
server_key_iv: "1c405f2be996063b14fe28bc4e154b67"
server_key_version: "encryptionkey@0"
I20260808 15:53:31.068106 23569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:53:31.072001 23588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:31.073372 23569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:31.073515 23569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "d0993dafdf0c4c2f81ed231df30fd8ca"
format_stamp: "Formatted at 2026-08-08 15:53:31 on dist-test-slave-twwt"
server_key: "1116214dec7f9671b903abe9697667c0"
server_key_iv: "1c405f2be996063b14fe28bc4e154b67"
server_key_version: "encryptionkey@0"
I20260808 15:53:31.073695 23569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:31.087698 23569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:31.088521 23569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:31.088800 23569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:31.089975 23569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:31.091799 23569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:31.091887 23569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:31.091964 23569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:31.092021 23569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:31.128511 23569 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:41495
I20260808 15:53:31.128618 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:41495 every 8 connection(s)
I20260808 15:53:31.129963 23569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:31.133364 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 23569
I20260808 15:53:31.133554 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:53:31.134222 27987 external_mini_cluster.cc:1428] Setting key 3b3c0b67c655bc5b932981c3435c4dea
I20260808 15:53:31.144100 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.132942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:53887 (local address 127.27.84.254:42307)
0808 15:53:31.133130 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.133134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.135120 (+ 1986us) server_negotiation.cc:407] Connection header received
0808 15:53:31.137238 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.137243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.137338 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.137455 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:31.138656 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.139441 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.141121 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.141536 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.141689 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.143220 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:31.143225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:31.143228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:31.143247 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:31.143281 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:31.143293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:31.143396 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.143650 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.143752 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:53:31.145303 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.133654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42307 (local address 127.27.84.195:53887)
0808 15:53:31.134925 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.134980 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:31.136932 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:31.137738 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:31.137748 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:31.137848 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:31.138449 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.138467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.139634 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.139638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.140948 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.140962 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.141809 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.141817 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.142064 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.142857 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:31.142859 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:31.142904 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:31.142989 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:31.143399 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:31.143413 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:31.143423 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:53:31.143638 (+ 215us) client_negotiation.cc:240] Negotiation successful
0808 15:53:31.144072 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":353,"threads_started":1}
I20260808 15:53:31.146977 23758 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42307
I20260808 15:53:31.147318 23758 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:31.148293 23758 heartbeater.cc:511] Master 127.27.84.254:42307 requested a full tablet report, sending...
I20260808 15:53:31.150118 23091 ts_manager.cc:194] Registered new tserver with Master: d0993dafdf0c4c2f81ed231df30fd8ca (127.27.84.195:41495)
I20260808 15:53:31.151378 23091 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.195:53887
I20260808 15:53:31.162873 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:53:31.173432 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.167306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.27.84.254:42307)
0808 15:53:31.167512 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.167516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.167890 (+ 374us) server_negotiation.cc:407] Connection header received
0808 15:53:31.168030 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.168034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.168106 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.168191 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:31.168725 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.169442 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.171115 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.171550 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.171647 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.172723 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:31.172728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:31.172731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:31.172751 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:31.172786 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:31.172798 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:31.172879 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.173080 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.173167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:53:31.174906 23361 heartbeater.cc:503] Master 127.27.84.254:42307 was elected leader, sending a full tablet report...
I20260808 15:53:31.178022 23091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:53:31.185091 23091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:53:31.221855 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.199599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.27.84.194:35333)
0808 15:53:31.213568 (+ 13969us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.213581 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.213620 (+ 39us) server_negotiation.cc:407] Connection header received
0808 15:53:31.213740 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.213753 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.214092 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.214348 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:31.215118 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.216702 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.219132 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.220315 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.220456 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.220716 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.221330 (+ 614us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.221515 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13797,"thread_start_us":13681,"threads_started":1}
I20260808 15:53:31.221889 23780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.199425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:35333 (local address 127.0.0.1:52712)
0808 15:53:31.211376 (+ 11951us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.211407 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:31.211596 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:31.214361 (+ 2765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:31.214368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:31.214399 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:31.214834 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.214854 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.216885 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.216890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.218940 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.218952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.220843 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.220849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.221064 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.221119 (+ 55us) client_negotiation.cc:769] Sending connection context
0808 15:53:31.221269 (+ 150us) client_negotiation.cc:240] Negotiation successful
0808 15:53:31.221478 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11731,"mutex_wait_us":4,"thread_start_us":12286,"threads_started":1}
I20260808 15:53:31.222754 23776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.198383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:41495 (local address 127.0.0.1:60078)
0808 15:53:31.199581 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.199625 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:31.199800 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:31.214807 (+ 15007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:31.214814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:31.214841 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:31.215402 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.215412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.217283 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.217291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.219895 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.219916 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.221909 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.221916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.222120 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.222149 (+ 29us) client_negotiation.cc:769] Sending connection context
0808 15:53:31.222210 (+ 61us) client_negotiation.cc:240] Negotiation successful
0808 15:53:31.222345 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":69,"thread_start_us":766,"threads_started":1}
I20260808 15:53:31.223317 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.198682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60078 (local address 127.27.84.195:41495)
0808 15:53:31.214187 (+ 15505us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.214206 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.214241 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:53:31.214342 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.214349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.214562 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.214804 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:31.215677 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.217134 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.220150 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.221654 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.222462 (+ 808us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.222765 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.222841 (+ 76us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.223005 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15326,"thread_start_us":15122,"threads_started":1}
I20260808 15:53:31.223439 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.199999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38864)
0808 15:53:31.213415 (+ 13416us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.213430 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:31.213584 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:31.214659 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:31.214665 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:31.214692 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:31.215394 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.215408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.217515 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.217521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.219925 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.219936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.222412 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.222419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.222580 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.222633 (+ 53us) client_negotiation.cc:769] Sending connection context
0808 15:53:31.222710 (+ 77us) client_negotiation.cc:240] Negotiation successful
0808 15:53:31.223067 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13301,"mutex_wait_us":109,"thread_start_us":13172,"threads_started":1}
I20260808 15:53:31.224393 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.200353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38864 (local address 127.27.84.193:39651)
0808 15:53:31.212715 (+ 12362us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.212729 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.213566 (+ 837us) server_negotiation.cc:407] Connection header received
0808 15:53:31.213832 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.213843 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.214130 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.214346 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:31.215846 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.217377 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.220154 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.221654 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.221808 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.223785 (+ 1977us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.223856 (+ 71us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.224028 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12087,"thread_start_us":12371,"threads_started":1}
I20260808 15:53:31.226580 23458 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 (DEFAULT_TABLE table=test-table [id=78b28d44d8c24ece9877bc593211d933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:31.227018 23664 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 (DEFAULT_TABLE table=test-table [id=78b28d44d8c24ece9877bc593211d933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:31.228225 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf44b13672c42c4932842a3ae7004f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:31.228231 23458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf44b13672c42c4932842a3ae7004f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:31.227844 23269 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 (DEFAULT_TABLE table=test-table [id=78b28d44d8c24ece9877bc593211d933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:31.229122 23269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf44b13672c42c4932842a3ae7004f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:31.240337 23785 tablet_bootstrap.cc:492] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Bootstrap starting.
I20260808 15:53:31.243443 23789 tablet_bootstrap.cc:492] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Bootstrap starting.
I20260808 15:53:31.244796 23785 tablet_bootstrap.cc:654] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:31.246376 23785 log.cc:824] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:31.248090 23790 tablet_bootstrap.cc:492] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Bootstrap starting.
I20260808 15:53:31.248396 23789 tablet_bootstrap.cc:654] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:31.250233 23789 log.cc:824] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:31.251865 23790 tablet_bootstrap.cc:654] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:31.253242 23790 log.cc:824] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:31.253508 23785 tablet_bootstrap.cc:492] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: No bootstrap required, opened a new log
I20260808 15:53:31.253777 23789 tablet_bootstrap.cc:492] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: No bootstrap required, opened a new log
I20260808 15:53:31.253844 23785 ts_tablet_manager.cc:1397] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260808 15:53:31.254112 23789 ts_tablet_manager.cc:1397] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260808 15:53:31.256646 23790 tablet_bootstrap.cc:492] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: No bootstrap required, opened a new log
I20260808 15:53:31.256963 23790 ts_tablet_manager.cc:1397] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 15:53:31.262347 23785 raft_consensus.cc:348] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.262871 23785 raft_consensus.cc:374] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:31.262975 23785 raft_consensus.cc:729] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0993dafdf0c4c2f81ed231df30fd8ca, State: Initialized, Role: FOLLOWER
I20260808 15:53:31.263648 23785 consensus_queue.cc:260] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.264844 23758 heartbeater.cc:503] Master 127.27.84.254:42307 was elected leader, sending a full tablet report...
I20260808 15:53:31.264974 23789 raft_consensus.cc:348] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.265508 23789 raft_consensus.cc:374] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:31.265594 23789 raft_consensus.cc:729] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3081f3f14f419f88c9ed32a6b3d617, State: Initialized, Role: FOLLOWER
I20260808 15:53:31.265647 23785 ts_tablet_manager.cc:1428] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260808 15:53:31.266206 23789 consensus_queue.cc:260] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.267446 23566 heartbeater.cc:503] Master 127.27.84.254:42307 was elected leader, sending a full tablet report...
I20260808 15:53:31.268108 23789 ts_tablet_manager.cc:1428] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260808 15:53:31.268419 23790 raft_consensus.cc:348] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.268843 23790 raft_consensus.cc:374] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:31.268987 23790 raft_consensus.cc:729] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ad09b17cb24c3594b82f72cb266421, State: Initialized, Role: FOLLOWER
I20260808 15:53:31.269671 23790 consensus_queue.cc:260] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.271543 23790 ts_tablet_manager.cc:1428] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260808 15:53:31.374696 23801 raft_consensus.cc:482] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:31.375079 23801 raft_consensus.cc:504] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.376993 23801 leader_election.cc:288] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0993dafdf0c4c2f81ed231df30fd8ca (127.27.84.195:41495), bd3081f3f14f419f88c9ed32a6b3d617 (127.27.84.194:35333)
W20260808 15:53:31.378540 23567 tablet.cc:2369] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:53:31.381903 23760 tablet.cc:2369] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:31.385023 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.377570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:35541 (local address 127.27.84.195:41495)
0808 15:53:31.377791 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.377797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.379166 (+ 1369us) server_negotiation.cc:407] Connection header received
0808 15:53:31.379478 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.379484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.379569 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.379623 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:31.380440 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.381145 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.383318 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.384101 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.384303 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.384376 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.384656 (+ 280us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.384780 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:53:31.385128 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.377541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:41495 (local address 127.27.84.193:35541)
0808 15:53:31.379050 (+ 1509us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.379077 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:31.379278 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:31.379775 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:31.379782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:31.379804 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:31.380249 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.380260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.381292 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.381306 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.383123 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.383136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.384310 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.384317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.384475 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.384502 (+ 27us) client_negotiation.cc:769] Sending connection context
0808 15:53:31.384612 (+ 110us) client_negotiation.cc:240] Negotiation successful
0808 15:53:31.384739 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":215,"thread_start_us":685,"threads_started":1}
I20260808 15:53:31.385149 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.377728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:50015 (local address 127.27.84.194:35333)
0808 15:53:31.378003 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.378011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.379357 (+ 1346us) server_negotiation.cc:407] Connection header received
0808 15:53:31.379454 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.379459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.379554 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.379675 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:31.380377 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.381070 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.383488 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.384212 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.384407 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.384482 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.384769 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.384893 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260808 15:53:31.385303 23803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.377535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:35333 (local address 127.27.84.193:50015)
0808 15:53:31.378850 (+ 1315us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.378878 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:31.379063 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:31.379694 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:31.379699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:31.379723 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:31.380208 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.380223 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.381213 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.381219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.383303 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:31.383317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.384454 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.384461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:31.384598 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.384620 (+ 22us) client_negotiation.cc:769] Sending connection context
0808 15:53:31.384728 (+ 108us) client_negotiation.cc:240] Negotiation successful
0808 15:53:31.384862 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":103,"spinlock_wait_cycles":23296,"thread_start_us":608,"threads_started":1}
I20260808 15:53:31.386253 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf44b13672c42c4932842a3ae7004f8" candidate_uuid: "69ad09b17cb24c3594b82f72cb266421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" is_pre_election: true
I20260808 15:53:31.386274 23489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf44b13672c42c4932842a3ae7004f8" candidate_uuid: "69ad09b17cb24c3594b82f72cb266421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" is_pre_election: true
I20260808 15:53:31.386812 23689 raft_consensus.cc:2454] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ad09b17cb24c3594b82f72cb266421 in term 0.
I20260808 15:53:31.386824 23489 raft_consensus.cc:2454] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ad09b17cb24c3594b82f72cb266421 in term 0.
I20260808 15:53:31.387514 23199 leader_election.cc:302] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 69ad09b17cb24c3594b82f72cb266421, bd3081f3f14f419f88c9ed32a6b3d617, d0993dafdf0c4c2f81ed231df30fd8ca; no voters:
I20260808 15:53:31.387907 23801 raft_consensus.cc:2781] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:53:31.388062 23801 raft_consensus.cc:482] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:53:31.388191 23801 raft_consensus.cc:3037] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:31.390666 23801 raft_consensus.cc:504] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
I20260808 15:53:31.391422 23801 leader_election.cc:288] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [CANDIDATE]: Term 1 election: Requested vote from peers d0993dafdf0c4c2f81ed231df30fd8ca (127.27.84.195:41495), bd3081f3f14f419f88c9ed32a6b3d617 (127.27.84.194:35333)
I20260808 15:53:31.391872 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf44b13672c42c4932842a3ae7004f8" candidate_uuid: "69ad09b17cb24c3594b82f72cb266421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca"
I20260808 15:53:31.392092 23689 raft_consensus.cc:3037] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:31.392005 23489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf44b13672c42c4932842a3ae7004f8" candidate_uuid: "69ad09b17cb24c3594b82f72cb266421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3081f3f14f419f88c9ed32a6b3d617"
I20260808 15:53:31.392325 23489 raft_consensus.cc:3037] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:31.394425 23689 raft_consensus.cc:2454] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ad09b17cb24c3594b82f72cb266421 in term 1.
I20260808 15:53:31.394937 23489 raft_consensus.cc:2454] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ad09b17cb24c3594b82f72cb266421 in term 1.
I20260808 15:53:31.394972 23202 leader_election.cc:302] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ad09b17cb24c3594b82f72cb266421, d0993dafdf0c4c2f81ed231df30fd8ca; no voters:
I20260808 15:53:31.395395 23801 raft_consensus.cc:2781] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:31.395905 23801 raft_consensus.cc:686] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 1 LEADER]: Becoming Leader. State: Replica: 69ad09b17cb24c3594b82f72cb266421, State: Running, Role: LEADER
I20260808 15:53:31.396507 23801 consensus_queue.cc:237] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } }
W20260808 15:53:31.401472 23362 tablet.cc:2369] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:31.402038 23090 catalog_manager.cc:5495] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ad09b17cb24c3594b82f72cb266421 (127.27.84.193). New cstate: current_term: 1 leader_uuid: "69ad09b17cb24c3594b82f72cb266421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ad09b17cb24c3594b82f72cb266421" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 39651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 } health_report { overall_health: UNKNOWN } } }
I20260808 15:53:31.460711 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.454765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.27.84.193:39651)
0808 15:53:31.455012 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.455019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.455115 (+ 96us) server_negotiation.cc:407] Connection header received
0808 15:53:31.455384 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.455388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.455489 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.455653 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:53:31.456056 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.456807 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.458339 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.458889 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.459037 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.459598 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:53:31.460113 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:53:31.460298 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.460356 (+ 58us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.460441 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260808 15:53:31.469928 23489 raft_consensus.cc:1261] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 1 FOLLOWER]: Refusing update from remote peer 69ad09b17cb24c3594b82f72cb266421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:31.469923 23689 raft_consensus.cc:1261] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 1 FOLLOWER]: Refusing update from remote peer 69ad09b17cb24c3594b82f72cb266421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:31.471071 23805 consensus_queue.cc:1035] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0993dafdf0c4c2f81ed231df30fd8ca" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 41495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:31.471501 23801 consensus_queue.cc:1035] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3081f3f14f419f88c9ed32a6b3d617" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:31.490803 23812 mvcc.cc:205] Tried to move back new op lower bound from 7316293269369069568 to 7316293269085200384. Current Snapshot: MvccSnapshot[applied={T|T < 7316293269369069568}]
I20260808 15:53:31.503410 23092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:31.503716 23092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:31.507928 23092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45caf07029ad46e6a70a36c63009602d: Sending DeleteTablet for 3 replicas of tablet 5bf44b13672c42c4932842a3ae7004f8
I20260808 15:53:31.509213 23664 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:31 UTC) from {username='slave'} at 127.0.0.1:60078
I20260808 15:53:31.509208 23458 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:31 UTC) from {username='slave'} at 127.0.0.1:52712
I20260808 15:53:31.509207 23269 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:31 UTC) from {username='slave'} at 127.0.0.1:38864
I20260808 15:53:31.509963 23268 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf44b13672c42c4932842a3ae7004f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:31 UTC) from {username='slave'} at 127.0.0.1:38864
I20260808 15:53:31.510738 23825 tablet_replica.cc:330] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: stopping tablet replica
I20260808 15:53:31.511016 23827 tablet_replica.cc:330] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: stopping tablet replica
I20260808 15:53:31.511523 23827 raft_consensus.cc:2229] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:31.511655 23824 tablet_replica.cc:330] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: stopping tablet replica
I20260808 15:53:31.511988 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 23181
I20260808 15:53:31.512115 23824 raft_consensus.cc:2229] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:53:31.512131 23825 raft_consensus.cc:2229] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:31.512656 23825 raft_consensus.cc:2258] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:31.512873 23824 raft_consensus.cc:2258] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:31.513413 23827 raft_consensus.cc:2258] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:31.515781 23827 ts_tablet_manager.cc:1905] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:31.515890 23824 ts_tablet_manager.cc:1905] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:31.522925 23827 ts_tablet_manager.cc:1918] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:31.523104 23827 log.cc:1196] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/5bf44b13672c42c4932842a3ae7004f8
I20260808 15:53:31.523680 23827 ts_tablet_manager.cc:1939] T 5bf44b13672c42c4932842a3ae7004f8 P d0993dafdf0c4c2f81ed231df30fd8ca: Deleting consensus metadata
I20260808 15:53:31.526026 23824 ts_tablet_manager.cc:1918] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:31.526089 23067 catalog_manager.cc:4841] TS d0993dafdf0c4c2f81ed231df30fd8ca (127.27.84.195:41495): tablet 5bf44b13672c42c4932842a3ae7004f8 (table test-table [id=78b28d44d8c24ece9877bc593211d933]) successfully deleted
I20260808 15:53:31.526224 23824 log.cc:1196] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/5bf44b13672c42c4932842a3ae7004f8
I20260808 15:53:31.528402 23825 ts_tablet_manager.cc:1905] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:31.534512 23825 ts_tablet_manager.cc:1918] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:31.534718 23825 log.cc:1196] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/5bf44b13672c42c4932842a3ae7004f8
I20260808 15:53:31.535313 23825 ts_tablet_manager.cc:1939] T 5bf44b13672c42c4932842a3ae7004f8 P bd3081f3f14f419f88c9ed32a6b3d617: Deleting consensus metadata
I20260808 15:53:31.535866 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.513697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.27.84.193:39651)
0808 15:53:31.514058 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.514064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.527214 (+ 13150us) server_negotiation.cc:407] Connection header received
0808 15:53:31.527446 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.527453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.527560 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.527683 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:31.528408 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.529607 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.531681 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.532410 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.532531 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.534072 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:31.534085 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:31.534104 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:31.534941 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:31.535011 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:31.535037 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:31.535160 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.535352 (+ 192us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.535474 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260808 15:53:31.536701 23354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:31.537335 23066 catalog_manager.cc:4841] TS bd3081f3f14f419f88c9ed32a6b3d617 (127.27.84.194:35333): tablet 5bf44b13672c42c4932842a3ae7004f8 (table test-table [id=78b28d44d8c24ece9877bc593211d933]) successfully deleted
I20260808 15:53:31.634634 23824 ts_tablet_manager.cc:1939] T 5bf44b13672c42c4932842a3ae7004f8 P 69ad09b17cb24c3594b82f72cb266421: Deleting consensus metadata
I20260808 15:53:31.635267 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 23181
W20260808 15:53:31.661057 23067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:39651 (error 108)
I20260808 15:53:31.661485 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 23366
I20260808 15:53:31.663484 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.662925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38896)
0808 15:53:31.663118 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.663298 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260808 15:53:31.663776 23067 catalog_manager.cc:4584] TS 69ad09b17cb24c3594b82f72cb266421 (127.27.84.193:39651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bf44b13672c42c4932842a3ae7004f8: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
I20260808 15:53:31.669341 23781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.662513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.27.84.194:35333)
0808 15:53:31.662743 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.662750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.662774 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:53:31.662989 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.662995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.663103 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.663224 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:31.663821 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.664715 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.666153 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.666579 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.666802 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.667849 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:31.667862 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:31.667881 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:31.668495 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:31.668544 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:31.668564 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:31.668708 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.668852 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.668965 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260808 15:53:31.670114 23557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:31.708734 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.708162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38900)
0808 15:53:31.708415 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.708537 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260808 15:53:31.709970 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.709553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38902)
0808 15:53:31.709729 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.709812 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260808 15:53:31.717368 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.716896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38912)
0808 15:53:31.717098 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.717187 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260808 15:53:31.718410 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.718081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38924)
0808 15:53:31.718225 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.718300 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260808 15:53:31.759119 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.758548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38928)
0808 15:53:31.758842 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.758947 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260808 15:53:31.760332 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.759910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38930)
0808 15:53:31.760123 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.760201 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260808 15:53:31.764180 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 23366
I20260808 15:53:31.792385 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 23569
I20260808 15:53:31.795202 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.794738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38938)
0808 15:53:31.794917 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.795016 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260808 15:53:31.796271 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.795875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38942)
0808 15:53:31.796050 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.796145 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 15:53:31.800328 23777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.793382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.27.84.195:41495)
0808 15:53:31.793697 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:31.793703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:31.793725 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:53:31.793898 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:31.793904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:31.794001 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:31.794117 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:31.794575 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.795446 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.797016 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:31.797502 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:31.797622 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:31.798874 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:31.798882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:31.798898 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:31.799486 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:31.799531 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:31.799551 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:31.799645 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:31.799754 (+ 109us) server_negotiation.cc:299] Negotiation successful
0808 15:53:31.799885 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260808 15:53:31.801113 23751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:31.861235 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.860646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38954)
0808 15:53:31.860941 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.861039 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260808 15:53:31.862357 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.861987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38956)
0808 15:53:31.862159 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.862228 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260808 15:53:31.892081 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 23569
I20260808 15:53:31.900226 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.899591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38962)
0808 15:53:31.899900 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.900007 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 15:53:31.901568 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:31.901191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38966)
0808 15:53:31.901348 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:31.901445 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:53:31.920739 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 23045
I20260808 15:53:31.921936 23135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:32.039374 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:32.038593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:39651 (local address 127.0.0.1:38972)
0808 15:53:32.038983 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:32.039139 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:39651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260808 15:53:32.040189 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 23045
2026-08-08T15:53:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3060 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:32 dist-test-slave-twwt krb5kdc[23838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:32 dist-test-slave-twwt krb5kdc[23838](info): set up 2 sockets
Aug 08 15:53:32 dist-test-slave-twwt krb5kdc[23838](info): commencing operation
krb5kdc: starting...
W20260808 15:53:33.417877 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:33 dist-test-slave-twwt krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:53:33.767251 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:45203
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:45203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:34.133876 23855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:34.134258 23855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:34.134435 23855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:34.141991 23855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:34.142164 23855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:34.142279 23855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:34.142385 23855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:34.142481 23855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:34.152743 23855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:45203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:45203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:34.154534 23855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:34.156005 23855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:34.168313 23866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:34.168399 23869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:34.168380 23864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:34.169974 23855 server_base.cc:1034] running on GCE node
I20260808 15:53:34.170874 23855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:34.172416 23855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:34.173631 23855 hybrid_clock.cc:648] HybridClock initialized: now 1786204414173573 us; error 51 us; skew 500 ppm
I20260808 15:53:34.173965 23855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:34 dist-test-slave-twwt krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:34.183063 23855 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:53:34.184724 23855 webserver.cc:460] Webserver started at http://127.27.84.254:34497/ using document root <none> and password file <none>
I20260808 15:53:34.185437 23855 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:34.185547 23855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:34.185817 23855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:34.188992 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "31215426dd2a451791779f8bc7dd7296"
format_stamp: "Formatted at 2026-08-08 15:53:34 on dist-test-slave-twwt"
server_key: "66cd6d8ee75137283d89f6d9afafef1d"
server_key_iv: "10e018528b26fef77e92b2de11d3d485"
server_key_version: "encryptionkey@0"
I20260808 15:53:34.189745 23855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "31215426dd2a451791779f8bc7dd7296"
format_stamp: "Formatted at 2026-08-08 15:53:34 on dist-test-slave-twwt"
server_key: "66cd6d8ee75137283d89f6d9afafef1d"
server_key_iv: "10e018528b26fef77e92b2de11d3d485"
server_key_version: "encryptionkey@0"
I20260808 15:53:34.195236 23855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:53:34.198886 23879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:34.200394 23855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:34.200542 23855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "31215426dd2a451791779f8bc7dd7296"
format_stamp: "Formatted at 2026-08-08 15:53:34 on dist-test-slave-twwt"
server_key: "66cd6d8ee75137283d89f6d9afafef1d"
server_key_iv: "10e018528b26fef77e92b2de11d3d485"
server_key_version: "encryptionkey@0"
I20260808 15:53:34.200726 23855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:34.221287 23855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:34.225973 23855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:34.226228 23855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:34.246354 23855 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:45203
I20260808 15:53:34.246376 23955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:45203 every 8 connection(s)
I20260808 15:53:34.247898 23855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:34.251829 23956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:34.253095 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 23855
I20260808 15:53:34.253312 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:34.253806 27987 external_mini_cluster.cc:1428] Setting key 4ce747a4cd7b1d0217a3dcf38585c537
I20260808 15:53:34.266983 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296: Bootstrap starting.
Aug 08 15:53:34 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:34.270252 23956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:34.271582 23956 log.cc:824] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:34.274684 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296: No bootstrap required, opened a new log
I20260808 15:53:34.279211 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:34.255973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.27.84.254:45203)
0808 15:53:34.256845 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:34.256861 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:34.256920 (+ 59us) server_negotiation.cc:407] Connection header received
0808 15:53:34.259044 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:34.259076 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:34.259624 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:34.260121 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:34.261569 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:34.263629 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:34.265049 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:34.266073 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:34.266258 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:34.270548 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:34.270556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:34.270572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:34.270679 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:34.274019 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:34.274889 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:34.274897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:34.274905 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:34.275134 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:34.275558 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:34.275565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:34.275573 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:34.276311 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:34.276616 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:34.277153 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 15:53:34.277655 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":354,"threads_started":1}
I20260808 15:53:34.283970 23956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31215426dd2a451791779f8bc7dd7296" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45203 } }
I20260808 15:53:34.284385 23956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:34.284510 23956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31215426dd2a451791779f8bc7dd7296, State: Initialized, Role: FOLLOWER
I20260808 15:53:34.285094 23956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31215426dd2a451791779f8bc7dd7296" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45203 } }
I20260808 15:53:34.285343 23956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:34.285471 23956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:34.285652 23956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:34.287958 23956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31215426dd2a451791779f8bc7dd7296" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45203 } }
I20260808 15:53:34.288475 23956 leader_election.cc:302] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31215426dd2a451791779f8bc7dd7296; no voters:
I20260808 15:53:34.289083 23956 leader_election.cc:288] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:34.289274 23963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:34.290083 23963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [term 1 LEADER]: Becoming Leader. State: Replica: 31215426dd2a451791779f8bc7dd7296, State: Running, Role: LEADER
I20260808 15:53:34.290658 23963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31215426dd2a451791779f8bc7dd7296" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45203 } }
I20260808 15:53:34.291498 23956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:34.293855 23970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31215426dd2a451791779f8bc7dd7296. Latest consensus state: current_term: 1 leader_uuid: "31215426dd2a451791779f8bc7dd7296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31215426dd2a451791779f8bc7dd7296" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45203 } } }
I20260808 15:53:34.294322 23970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:34.294432 23967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31215426dd2a451791779f8bc7dd7296" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31215426dd2a451791779f8bc7dd7296" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45203 } } }
I20260808 15:53:34.294850 23967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:34.296445 23978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:34.301751 23978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:34.311846 23978 catalog_manager.cc:1347] Generated new cluster ID: b8e10456fcea4e259f5540e7db07449c
I20260808 15:53:34.311991 23978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:34.323388 23978 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:34.324786 23978 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:34.331111 23978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296: Generated new TSK 0
I20260808 15:53:34.332223 23978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:53:34.423070 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45203
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:34.792657 23996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:34.793028 23996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:34.793196 23996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:34.800966 23996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:34.801137 23996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:34.801321 23996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:34.811676 23996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:34.813122 23996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:34.814553 23996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:34.830185 24007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:34.830287 24005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:34.830228 24010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:34.836388 23996 server_base.cc:1034] running on GCE node
I20260808 15:53:34.837194 23996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:34.838382 23996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:34.839648 23996 hybrid_clock.cc:648] HybridClock initialized: now 1786204414839549 us; error 97 us; skew 500 ppm
I20260808 15:53:34.840088 23996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:34 dist-test-slave-twwt krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:34.843761 23996 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:53:34.845643 23996 webserver.cc:460] Webserver started at http://127.27.84.193:35691/ using document root <none> and password file <none>
I20260808 15:53:34.846807 23996 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:34.846972 23996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:34.847319 23996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:34.850339 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "956fb2343374406fa33afb459b522c81"
format_stamp: "Formatted at 2026-08-08 15:53:34 on dist-test-slave-twwt"
server_key: "489703d1c4f020624d6aac45e666e8e7"
server_key_iv: "75e341e184bc21417bec3c708b054962"
server_key_version: "encryptionkey@0"
I20260808 15:53:34.851200 23996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "956fb2343374406fa33afb459b522c81"
format_stamp: "Formatted at 2026-08-08 15:53:34 on dist-test-slave-twwt"
server_key: "489703d1c4f020624d6aac45e666e8e7"
server_key_iv: "75e341e184bc21417bec3c708b054962"
server_key_version: "encryptionkey@0"
I20260808 15:53:34.856783 23996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:53:34.860630 24023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:34.862087 23996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:53:34.862293 23996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "956fb2343374406fa33afb459b522c81"
format_stamp: "Formatted at 2026-08-08 15:53:34 on dist-test-slave-twwt"
server_key: "489703d1c4f020624d6aac45e666e8e7"
server_key_iv: "75e341e184bc21417bec3c708b054962"
server_key_version: "encryptionkey@0"
I20260808 15:53:34.862527 23996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:34.880818 23996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:34.885321 23996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:34.885646 23996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:34.886771 23996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:34.888281 23996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:34.888396 23996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:34.888553 23996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:34.888631 23996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:34.925472 23996 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:37783
I20260808 15:53:34.925570 24204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:37783 every 8 connection(s)
I20260808 15:53:34.926882 23996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:34.930053 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 23996
I20260808 15:53:34.930260 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:53:34.930933 27987 external_mini_cluster.cc:1428] Setting key 62bd29fbeeda0a486740866fcc4cc2cd
Aug 08 15:53:34 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:34.948403 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:34.929838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55309 (local address 127.27.84.254:45203)
0808 15:53:34.930398 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:34.930403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:34.931672 (+ 1269us) server_negotiation.cc:407] Connection header received
0808 15:53:34.933605 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:34.933611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:34.933709 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:34.933844 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:34.935485 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:34.936218 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:34.937848 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:34.938256 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:34.938410 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:34.942383 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:34.942389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:34.942392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:34.942425 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:34.945127 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:34.945896 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:34.945904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:34.945907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:34.946034 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:34.946499 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:34.946504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:34.946512 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:34.946876 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:34.947046 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:34.947778 (+ 732us) server_negotiation.cc:299] Negotiation successful
0808 15:53:34.948020 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":283,"threads_started":1}
I20260808 15:53:34.949419 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:34.930573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:45203 (local address 127.27.84.193:55309)
0808 15:53:34.931494 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:34.931542 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:34.933322 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:34.934104 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:34.934113 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:34.934722 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:34.935308 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:34.935325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:34.936403 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:34.936413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:34.937674 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:34.937686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:34.938523 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:34.938530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:34.938820 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:34.939541 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:34.939570 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:34.942135 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:34.945284 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:34.945295 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:34.945311 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:34.945750 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:34.946157 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:34.946162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:34.946166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:34.946369 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:34.947017 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:34.947026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:34.947385 (+ 359us) client_negotiation.cc:769] Sending connection context
0808 15:53:34.947661 (+ 276us) client_negotiation.cc:240] Negotiation successful
0808 15:53:34.948226 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":392,"threads_started":1}
I20260808 15:53:34.951212 24206 heartbeater.cc:347] Connected to a master server at 127.27.84.254:45203
I20260808 15:53:34.951541 24206 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:34.952313 24206 heartbeater.cc:511] Master 127.27.84.254:45203 requested a full tablet report, sending...
I20260808 15:53:34.954945 23909 ts_manager.cc:194] Registered new tserver with Master: 956fb2343374406fa33afb459b522c81 (127.27.84.193:37783)
I20260808 15:53:34.957372 23909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:55309
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:53:35.028626 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45203
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:35.393903 24215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:35.394224 24215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:35.394325 24215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:35.401834 24215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:35.401957 24215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:35.402076 24215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:35.412325 24215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:35.413790 24215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:35.415242 24215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:35.429929 24222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:35.429994 24225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:35.430030 24221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:35.432478 24215 server_base.cc:1034] running on GCE node
I20260808 15:53:35.433207 24215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:35.434242 24215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:35.435484 24215 hybrid_clock.cc:648] HybridClock initialized: now 1786204415435432 us; error 47 us; skew 500 ppm
I20260808 15:53:35.435858 24215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:35 dist-test-slave-twwt krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:35.439121 24215 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:53:35.440760 24215 webserver.cc:460] Webserver started at http://127.27.84.194:45181/ using document root <none> and password file <none>
I20260808 15:53:35.441478 24215 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:35.441582 24215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:35.441864 24215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:35.444954 24215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "ca9988b631ae4491a958f4da52c528b9"
format_stamp: "Formatted at 2026-08-08 15:53:35 on dist-test-slave-twwt"
server_key: "7abb7367f52083f197b3f074e15bfbb2"
server_key_iv: "e19a078a45696b3b0ef8673f5f439fa0"
server_key_version: "encryptionkey@0"
I20260808 15:53:35.445739 24215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "ca9988b631ae4491a958f4da52c528b9"
format_stamp: "Formatted at 2026-08-08 15:53:35 on dist-test-slave-twwt"
server_key: "7abb7367f52083f197b3f074e15bfbb2"
server_key_iv: "e19a078a45696b3b0ef8673f5f439fa0"
server_key_version: "encryptionkey@0"
I20260808 15:53:35.451460 24215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:53:35.455145 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:35.456842 24215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:53:35.457010 24215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "ca9988b631ae4491a958f4da52c528b9"
format_stamp: "Formatted at 2026-08-08 15:53:35 on dist-test-slave-twwt"
server_key: "7abb7367f52083f197b3f074e15bfbb2"
server_key_iv: "e19a078a45696b3b0ef8673f5f439fa0"
server_key_version: "encryptionkey@0"
I20260808 15:53:35.457206 24215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:35.480062 24215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:35.484810 24215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:35.485069 24215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:35.486142 24215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:35.487545 24215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:35.487622 24215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:35.487712 24215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:35.487775 24215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:35.523689 24215 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:40239
I20260808 15:53:35.523778 24408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:40239 every 8 connection(s)
I20260808 15:53:35.525198 24215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:35.535122 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 24215
I20260808 15:53:35.535358 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:53:35.536051 27987 external_mini_cluster.cc:1428] Setting key 5091594ddf0aa9dbbd99da5ecb71d198
Aug 08 15:53:35 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:35.548796 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:35.528085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:41835 (local address 127.27.84.254:45203)
0808 15:53:35.528535 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:35.528539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:35.529671 (+ 1132us) server_negotiation.cc:407] Connection header received
0808 15:53:35.532295 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:35.532300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:35.532387 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:35.532466 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:35.534603 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:35.535523 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:35.537596 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:35.538088 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:35.538242 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:35.542255 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:35.542261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:35.542264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:35.542299 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:35.545192 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:35.546063 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:35.546069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:35.546073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:35.546225 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:35.546788 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:35.546796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:35.546799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:35.547118 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:35.547308 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:35.548118 (+ 810us) server_negotiation.cc:299] Negotiation successful
0808 15:53:35.548388 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":221,"threads_started":1}
I20260808 15:53:35.550050 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:35.528711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:45203 (local address 127.27.84.194:41835)
0808 15:53:35.529497 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:35.529536 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:35.531953 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:35.533024 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:35.533036 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:35.533820 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:35.534422 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:35.534440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:35.535696 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:35.535702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:35.537430 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:35.537444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:35.538345 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:35.538353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:35.538661 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:35.539453 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:35.539481 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:35.541990 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:35.545392 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:35.545402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:35.545422 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:35.545917 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:35.546369 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:35.546374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:35.546379 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:35.546596 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:35.547274 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:35.547283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:35.547666 (+ 383us) client_negotiation.cc:769] Sending connection context
0808 15:53:35.547984 (+ 318us) client_negotiation.cc:240] Negotiation successful
0808 15:53:35.548602 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":285,"threads_started":1}
I20260808 15:53:35.551652 24409 heartbeater.cc:347] Connected to a master server at 127.27.84.254:45203
I20260808 15:53:35.552011 24409 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:35.552943 24409 heartbeater.cc:511] Master 127.27.84.254:45203 requested a full tablet report, sending...
I20260808 15:53:35.554719 23909 ts_manager.cc:194] Registered new tserver with Master: ca9988b631ae4491a958f4da52c528b9 (127.27.84.194:40239)
I20260808 15:53:35.555737 23909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:41835
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:53:35.614715 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45203
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:44253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:53:35.961339 24206 heartbeater.cc:503] Master 127.27.84.254:45203 was elected leader, sending a full tablet report...
W20260808 15:53:35.985656 24418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:35.985983 24418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:35.986089 24418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:35.994045 24418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:35.994170 24418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:35.994288 24418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:36.004915 24418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:44253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:36.006362 24418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:36.007822 24418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:36.022435 24424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:36.022467 24428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:36.022504 24425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:36.023463 24418 server_base.cc:1034] running on GCE node
I20260808 15:53:36.024255 24418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:36.025425 24418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:36.026682 24418 hybrid_clock.cc:648] HybridClock initialized: now 1786204416026577 us; error 54 us; skew 500 ppm
I20260808 15:53:36.027019 24418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:36 dist-test-slave-twwt krb5kdc[23838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:36.030305 24418 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:53:36.032025 24418 webserver.cc:460] Webserver started at http://127.27.84.195:34029/ using document root <none> and password file <none>
I20260808 15:53:36.032790 24418 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:36.032940 24418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:36.033294 24418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:36.036504 24418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "538ae1257daa4810a998a93cea99e8aa"
format_stamp: "Formatted at 2026-08-08 15:53:36 on dist-test-slave-twwt"
server_key: "26421d0639cf8cdd91c6cee8864b9ae3"
server_key_iv: "65cf4b94059afc7cc7f56b34005ed72b"
server_key_version: "encryptionkey@0"
I20260808 15:53:36.037348 24418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "538ae1257daa4810a998a93cea99e8aa"
format_stamp: "Formatted at 2026-08-08 15:53:36 on dist-test-slave-twwt"
server_key: "26421d0639cf8cdd91c6cee8864b9ae3"
server_key_iv: "65cf4b94059afc7cc7f56b34005ed72b"
server_key_version: "encryptionkey@0"
I20260808 15:53:36.043233 24418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260808 15:53:36.047098 24439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:36.048599 24418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:36.048796 24418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "538ae1257daa4810a998a93cea99e8aa"
format_stamp: "Formatted at 2026-08-08 15:53:36 on dist-test-slave-twwt"
server_key: "26421d0639cf8cdd91c6cee8864b9ae3"
server_key_iv: "65cf4b94059afc7cc7f56b34005ed72b"
server_key_version: "encryptionkey@0"
I20260808 15:53:36.049021 24418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:36.077678 24418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:36.082258 24418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:36.082648 24418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:36.083771 24418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:36.085335 24418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:36.085413 24418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:36.085554 24418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:36.085609 24418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:36.121968 24418 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:40371
I20260808 15:53:36.122061 24610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:40371 every 8 connection(s)
I20260808 15:53:36.123376 24418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:36.131346 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 24418
I20260808 15:53:36.131572 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:53:36.132083 27987 external_mini_cluster.cc:1428] Setting key 0c68372c13e5a6f7bbece4c2ac61b0c9
Aug 08 15:53:36 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:36.144682 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.126267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:43285 (local address 127.27.84.254:45203)
0808 15:53:36.126804 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.126811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.127727 (+ 916us) server_negotiation.cc:407] Connection header received
0808 15:53:36.129665 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.129671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.129767 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.129897 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:36.131562 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.132511 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.134560 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.135079 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.135246 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.138886 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:36.138891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:36.138894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:36.138927 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:36.141742 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.142464 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.142468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.142471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.142594 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.143097 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.143100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.143102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.143378 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:36.143496 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.144214 (+ 718us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.144396 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":282,"threads_started":1}
I20260808 15:53:36.145804 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.126802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:45203 (local address 127.27.84.195:43285)
0808 15:53:36.127552 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:36.127592 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:36.129378 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:36.130149 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:36.130157 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:36.130778 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:36.131381 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.131400 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.132694 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.132708 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.134422 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.134437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.135316 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.135322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.135618 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.136277 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:36.136300 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:36.138656 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:36.141883 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:36.141890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:36.141904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:36.142346 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:36.142749 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:36.142790 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:36.142793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:36.142986 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:36.143487 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:36.143493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:36.143861 (+ 368us) client_negotiation.cc:769] Sending connection context
0808 15:53:36.144095 (+ 234us) client_negotiation.cc:240] Negotiation successful
0808 15:53:36.144502 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":278,"threads_started":1}
I20260808 15:53:36.147382 24612 heartbeater.cc:347] Connected to a master server at 127.27.84.254:45203
I20260808 15:53:36.147720 24612 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:36.148599 24612 heartbeater.cc:511] Master 127.27.84.254:45203 requested a full tablet report, sending...
I20260808 15:53:36.150097 23905 ts_manager.cc:194] Registered new tserver with Master: 538ae1257daa4810a998a93cea99e8aa (127.27.84.195:40371)
I20260808 15:53:36.151379 23905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:43285
I20260808 15:53:36.160881 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:53:36.179790 24615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.165321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.27.84.254:45203)
0808 15:53:36.165552 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.165560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.166062 (+ 502us) server_negotiation.cc:407] Connection header received
0808 15:53:36.166270 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.166275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.166359 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.166470 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:36.168050 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.168842 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.170190 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.170719 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.170855 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.173786 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:36.173793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:36.173798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:36.173843 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:36.176604 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.177492 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.177498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.177502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.177640 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.178177 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.178182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.178185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.178526 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:36.178860 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.179281 (+ 421us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.179500 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:53:36.184373 23905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:53:36.191627 23905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:53:36.216714 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.205430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.27.84.194:40239)
0808 15:53:36.207470 (+ 2040us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.207482 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.207513 (+ 31us) server_negotiation.cc:407] Connection header received
0808 15:53:36.207624 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.207633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.207928 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.208769 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:36.209176 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.210786 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.213772 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.215166 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.215307 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.215514 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.216199 (+ 685us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.216370 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1836,"thread_start_us":1691,"threads_started":1}
I20260808 15:53:36.216894 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.205348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:40371 (local address 127.0.0.1:34890)
0808 15:53:36.207056 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:36.207078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:36.207197 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:36.209635 (+ 2438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:36.209639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:36.209665 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:36.210067 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.210081 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.212300 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.212305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.214036 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.214056 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.215946 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.215952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.216131 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.216164 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:53:36.216259 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:53:36.216412 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1590,"mutex_wait_us":3,"thread_start_us":1456,"threads_started":1}
I20260808 15:53:36.217341 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.205198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:40239 (local address 127.0.0.1:57432)
0808 15:53:36.206162 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:36.206238 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:36.206455 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:36.208221 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:36.208229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:36.208278 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:36.208907 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.208925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.210919 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.210927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.213588 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.213600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.215651 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.215657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.215894 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.215951 (+ 57us) client_negotiation.cc:769] Sending connection context
0808 15:53:36.216096 (+ 145us) client_negotiation.cc:240] Negotiation successful
0808 15:53:36.216287 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":26,"thread_start_us":489,"threads_started":1}
I20260808 15:53:36.217432 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.206018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.27.84.195:40371)
0808 15:53:36.209173 (+ 3155us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.209179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.209206 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:53:36.209277 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.209283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.209454 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.209647 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:36.210283 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.212132 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.214240 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.215726 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.215859 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.216086 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.216560 (+ 474us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.216753 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":1728,"threads_started":1}
I20260808 15:53:36.219566 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.207572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37783 (local address 127.0.0.1:59746)
0808 15:53:36.208116 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:36.208133 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:36.208253 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:36.211030 (+ 2777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:36.211035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:36.211067 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:36.211539 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.211551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.213500 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.213506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.217224 (+ 3718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.217237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.218788 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.218795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.218977 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.219010 (+ 33us) client_negotiation.cc:769] Sending connection context
0808 15:53:36.219110 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:53:36.219256 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":379,"threads_started":1}
I20260808 15:53:36.219712 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.207893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.27.84.193:37783)
0808 15:53:36.210408 (+ 2515us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.210417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.210447 (+ 30us) server_negotiation.cc:407] Connection header received
0808 15:53:36.210545 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.210553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.210811 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.211008 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:36.211798 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.213334 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.217401 (+ 4067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.218502 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.218675 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.218916 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.219178 (+ 262us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.219369 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2328,"thread_start_us":2145,"threads_started":1}
I20260808 15:53:36.221698 24516 tablet_service.cc:1467] Processing CreateTablet for tablet 6b75f829b59c458cbe41e9f4c3cda2a5 (DEFAULT_TABLE table=test-table [id=b1fc3e6e4ee647b09711399040606c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:36.221707 24313 tablet_service.cc:1467] Processing CreateTablet for tablet 6b75f829b59c458cbe41e9f4c3cda2a5 (DEFAULT_TABLE table=test-table [id=b1fc3e6e4ee647b09711399040606c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:36.223178 24516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b75f829b59c458cbe41e9f4c3cda2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:36.223208 24313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b75f829b59c458cbe41e9f4c3cda2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:36.223178 24105 tablet_service.cc:1467] Processing CreateTablet for tablet 6b75f829b59c458cbe41e9f4c3cda2a5 (DEFAULT_TABLE table=test-table [id=b1fc3e6e4ee647b09711399040606c08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:36.224651 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b75f829b59c458cbe41e9f4c3cda2a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:36.235373 24636 tablet_bootstrap.cc:492] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Bootstrap starting.
I20260808 15:53:36.238678 24637 tablet_bootstrap.cc:492] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Bootstrap starting.
I20260808 15:53:36.239034 24636 tablet_bootstrap.cc:654] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:36.240450 24641 tablet_bootstrap.cc:492] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Bootstrap starting.
I20260808 15:53:36.241993 24636 log.cc:824] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:36.242460 24637 tablet_bootstrap.cc:654] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:36.243548 24641 tablet_bootstrap.cc:654] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:36.244024 24637 log.cc:824] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:36.244575 24641 log.cc:824] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:36.247560 24637 tablet_bootstrap.cc:492] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: No bootstrap required, opened a new log
I20260808 15:53:36.247678 24636 tablet_bootstrap.cc:492] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: No bootstrap required, opened a new log
I20260808 15:53:36.247818 24637 ts_tablet_manager.cc:1397] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:53:36.247956 24636 ts_tablet_manager.cc:1397] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260808 15:53:36.248631 24641 tablet_bootstrap.cc:492] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: No bootstrap required, opened a new log
I20260808 15:53:36.248889 24641 ts_tablet_manager.cc:1397] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260808 15:53:36.258536 24637 raft_consensus.cc:348] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.259054 24637 raft_consensus.cc:374] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:36.259158 24637 raft_consensus.cc:729] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9988b631ae4491a958f4da52c528b9, State: Initialized, Role: FOLLOWER
I20260808 15:53:36.259629 24636 raft_consensus.cc:348] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.259895 24637 consensus_queue.cc:260] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.260076 24636 raft_consensus.cc:374] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:36.259876 24641 raft_consensus.cc:348] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.260176 24636 raft_consensus.cc:729] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538ae1257daa4810a998a93cea99e8aa, State: Initialized, Role: FOLLOWER
I20260808 15:53:36.260308 24641 raft_consensus.cc:374] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:36.260411 24641 raft_consensus.cc:729] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 956fb2343374406fa33afb459b522c81, State: Initialized, Role: FOLLOWER
I20260808 15:53:36.260808 24636 consensus_queue.cc:260] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.261205 24641 consensus_queue.cc:260] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.267349 24641 ts_tablet_manager.cc:1428] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Time spent starting tablet: real 0.018s user 0.010s sys 0.008s
I20260808 15:53:36.268146 24637 ts_tablet_manager.cc:1428] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260808 15:53:36.268483 24409 heartbeater.cc:503] Master 127.27.84.254:45203 was elected leader, sending a full tablet report...
I20260808 15:53:36.272094 24612 heartbeater.cc:503] Master 127.27.84.254:45203 was elected leader, sending a full tablet report...
I20260808 15:53:36.272831 24636 ts_tablet_manager.cc:1428] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Time spent starting tablet: real 0.025s user 0.024s sys 0.000s
W20260808 15:53:36.277271 24410 tablet.cc:2369] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:36.295456 24646 raft_consensus.cc:482] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:36.295804 24646 raft_consensus.cc:504] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.297747 24646 leader_election.cc:288] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538ae1257daa4810a998a93cea99e8aa (127.27.84.195:40371), 956fb2343374406fa33afb459b522c81 (127.27.84.193:37783)
I20260808 15:53:36.307000 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.298111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:46537 (local address 127.27.84.195:40371)
0808 15:53:36.298380 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.298388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.298986 (+ 598us) server_negotiation.cc:407] Connection header received
0808 15:53:36.299181 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.299186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.299258 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.299355 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:36.300180 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.301377 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.304352 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.305722 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.305890 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.305971 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.306543 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.306766 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:53:36.307000 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.298307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:49409 (local address 127.27.84.193:37783)
0808 15:53:36.298576 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.298583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.298923 (+ 340us) server_negotiation.cc:407] Connection header received
0808 15:53:36.299152 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.299158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.299241 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.299355 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:36.300180 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.301377 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.304349 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.305700 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.305889 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.305971 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.306543 (+ 572us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.306725 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 15:53:36.307264 24651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.298073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:40371 (local address 127.27.84.194:46537)
0808 15:53:36.298875 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:36.298892 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:36.299048 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:36.299390 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:36.299395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:36.299425 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:36.300003 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.300012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.301520 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.301525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.304149 (+ 2624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.304170 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.306005 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.306013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.306290 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.306367 (+ 77us) client_negotiation.cc:769] Sending connection context
0808 15:53:36.306470 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:53:36.306676 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":190,"spinlock_wait_cycles":25344,"thread_start_us":464,"threads_started":1}
I20260808 15:53:36.307289 24650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.298002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:37783 (local address 127.27.84.194:49409)
0808 15:53:36.298798 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:36.298818 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:36.298984 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:36.299390 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:36.299396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:36.299450 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:36.299998 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.300012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.301522 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.301526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.304161 (+ 2635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:36.304169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.306013 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.306018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:36.306290 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.306375 (+ 85us) client_negotiation.cc:769] Sending connection context
0808 15:53:36.306470 (+ 95us) client_negotiation.cc:240] Negotiation successful
0808 15:53:36.306676 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":194,"spinlock_wait_cycles":28416,"thread_start_us":368,"threads_started":1}
I20260808 15:53:36.308403 24133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b75f829b59c458cbe41e9f4c3cda2a5" candidate_uuid: "ca9988b631ae4491a958f4da52c528b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956fb2343374406fa33afb459b522c81" is_pre_election: true
I20260808 15:53:36.308475 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b75f829b59c458cbe41e9f4c3cda2a5" candidate_uuid: "ca9988b631ae4491a958f4da52c528b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538ae1257daa4810a998a93cea99e8aa" is_pre_election: true
I20260808 15:53:36.308998 24133 raft_consensus.cc:2454] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9988b631ae4491a958f4da52c528b9 in term 0.
I20260808 15:53:36.309202 24541 raft_consensus.cc:2454] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9988b631ae4491a958f4da52c528b9 in term 0.
I20260808 15:53:36.309785 24243 leader_election.cc:302] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 538ae1257daa4810a998a93cea99e8aa, 956fb2343374406fa33afb459b522c81, ca9988b631ae4491a958f4da52c528b9; no voters:
I20260808 15:53:36.310303 24646 raft_consensus.cc:2781] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:53:36.310462 24646 raft_consensus.cc:482] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:53:36.310585 24646 raft_consensus.cc:3037] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:36.313185 24646 raft_consensus.cc:504] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.313810 24646 leader_election.cc:288] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [CANDIDATE]: Term 1 election: Requested vote from peers 538ae1257daa4810a998a93cea99e8aa (127.27.84.195:40371), 956fb2343374406fa33afb459b522c81 (127.27.84.193:37783)
I20260808 15:53:36.314414 24541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b75f829b59c458cbe41e9f4c3cda2a5" candidate_uuid: "ca9988b631ae4491a958f4da52c528b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538ae1257daa4810a998a93cea99e8aa"
I20260808 15:53:36.314414 24133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b75f829b59c458cbe41e9f4c3cda2a5" candidate_uuid: "ca9988b631ae4491a958f4da52c528b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "956fb2343374406fa33afb459b522c81"
I20260808 15:53:36.314683 24541 raft_consensus.cc:3037] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:36.314670 24133 raft_consensus.cc:3037] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:36.318049 24133 raft_consensus.cc:2454] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9988b631ae4491a958f4da52c528b9 in term 1.
I20260808 15:53:36.318053 24541 raft_consensus.cc:2454] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9988b631ae4491a958f4da52c528b9 in term 1.
I20260808 15:53:36.318583 24241 leader_election.cc:302] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 538ae1257daa4810a998a93cea99e8aa, 956fb2343374406fa33afb459b522c81, ca9988b631ae4491a958f4da52c528b9; no voters:
I20260808 15:53:36.319020 24646 raft_consensus.cc:2781] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:36.319550 24646 raft_consensus.cc:686] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 1 LEADER]: Becoming Leader. State: Replica: ca9988b631ae4491a958f4da52c528b9, State: Running, Role: LEADER
I20260808 15:53:36.320097 24646 consensus_queue.cc:237] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } }
I20260808 15:53:36.325234 23903 catalog_manager.cc:5495] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca9988b631ae4491a958f4da52c528b9 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "ca9988b631ae4491a958f4da52c528b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9988b631ae4491a958f4da52c528b9" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 40239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 } health_report { overall_health: UNKNOWN } } }
I20260808 15:53:36.349104 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.343809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.27.84.194:40239)
0808 15:53:36.344004 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.344008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.344027 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:53:36.344116 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.344120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.344204 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.344335 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:53:36.344729 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.345482 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.346897 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.347316 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.347423 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.347920 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:53:36.348423 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:53:36.348564 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.348692 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.348802 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:53:36.358209 24133 raft_consensus.cc:1261] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 1 FOLLOWER]: Refusing update from remote peer ca9988b631ae4491a958f4da52c528b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:36.358208 24541 raft_consensus.cc:1261] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 1 FOLLOWER]: Refusing update from remote peer ca9988b631ae4491a958f4da52c528b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:36.359429 24652 consensus_queue.cc:1035] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "956fb2343374406fa33afb459b522c81" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 37783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:36.359839 24646 consensus_queue.cc:1035] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "538ae1257daa4810a998a93cea99e8aa" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:36.370047 24658 mvcc.cc:205] Tried to move back new op lower bound from 7316293289391603712 to 7316293289251508224. Current Snapshot: MvccSnapshot[applied={T|T < 7316293289391603712}]
W20260808 15:53:36.375221 24613 tablet.cc:2369] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:36.381518 24659 mvcc.cc:205] Tried to move back new op lower bound from 7316293289391603712 to 7316293289251508224. Current Snapshot: MvccSnapshot[applied={T|T < 7316293289391603712}]
I20260808 15:53:36.388005 23905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:36.388305 23905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:36.392432 23905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31215426dd2a451791779f8bc7dd7296: Sending DeleteTablet for 3 replicas of tablet 6b75f829b59c458cbe41e9f4c3cda2a5
I20260808 15:53:36.393787 24313 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b75f829b59c458cbe41e9f4c3cda2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:36 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:57432
I20260808 15:53:36.394229 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b75f829b59c458cbe41e9f4c3cda2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:36 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:59746
I20260808 15:53:36.394572 24516 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b75f829b59c458cbe41e9f4c3cda2a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:36 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:34890
I20260808 15:53:36.396140 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 23996
I20260808 15:53:36.409895 24675 tablet_replica.cc:330] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: stopping tablet replica
I20260808 15:53:36.410441 24675 raft_consensus.cc:2229] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:36.411087 24675 raft_consensus.cc:2258] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:36.413591 24675 ts_tablet_manager.cc:1905] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:36.416496 24674 tablet_replica.cc:330] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: stopping tablet replica
I20260808 15:53:36.417110 24677 tablet_replica.cc:330] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: stopping tablet replica
I20260808 15:53:36.417626 24677 raft_consensus.cc:2229] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:36.418219 24677 raft_consensus.cc:2258] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:36.419121 24674 raft_consensus.cc:2229] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:53:36.420058 24674 raft_consensus.cc:2258] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:36.421463 24675 ts_tablet_manager.cc:1918] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:36.421654 24675 log.cc:1196] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/6b75f829b59c458cbe41e9f4c3cda2a5
I20260808 15:53:36.422271 24675 ts_tablet_manager.cc:1939] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 956fb2343374406fa33afb459b522c81: Deleting consensus metadata
I20260808 15:53:36.425009 23884 catalog_manager.cc:4841] TS 956fb2343374406fa33afb459b522c81 (127.27.84.193:37783): tablet 6b75f829b59c458cbe41e9f4c3cda2a5 (table test-table [id=b1fc3e6e4ee647b09711399040606c08]) successfully deleted
Aug 08 15:53:36 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:53:36.427022 24677 ts_tablet_manager.cc:1905] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:36.427763 24674 ts_tablet_manager.cc:1905] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:36.435328 24677 ts_tablet_manager.cc:1918] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:36.435609 24677 log.cc:1196] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/6b75f829b59c458cbe41e9f4c3cda2a5
I20260808 15:53:36.435765 24674 ts_tablet_manager.cc:1918] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:36.435948 24674 log.cc:1196] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/6b75f829b59c458cbe41e9f4c3cda2a5
I20260808 15:53:36.436295 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.397795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.27.84.193:37783)
0808 15:53:36.398026 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.398035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.409496 (+ 11461us) server_negotiation.cc:407] Connection header received
0808 15:53:36.410017 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.410022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.410104 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.410190 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:36.412827 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.413932 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.420546 (+ 6614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.421323 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.421457 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.428207 (+ 6750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:36.428225 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:36.428254 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:36.428304 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:36.432329 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.433354 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.433364 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.433368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.433560 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.434075 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.434080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.434083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.434593 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:36.434947 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.435584 (+ 637us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.435838 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:53:36.436576 24674 ts_tablet_manager.cc:1939] T 6b75f829b59c458cbe41e9f4c3cda2a5 P ca9988b631ae4491a958f4da52c528b9: Deleting consensus metadata
I20260808 15:53:36.436621 24677 ts_tablet_manager.cc:1939] T 6b75f829b59c458cbe41e9f4c3cda2a5 P 538ae1257daa4810a998a93cea99e8aa: Deleting consensus metadata
I20260808 15:53:36.438048 24198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:36.438985 23881 catalog_manager.cc:4841] TS ca9988b631ae4491a958f4da52c528b9 (127.27.84.194:40239): tablet 6b75f829b59c458cbe41e9f4c3cda2a5 (table test-table [id=b1fc3e6e4ee647b09711399040606c08]) successfully deleted
I20260808 15:53:36.439333 23882 catalog_manager.cc:4841] TS 538ae1257daa4810a998a93cea99e8aa (127.27.84.195:40371): tablet 6b75f829b59c458cbe41e9f4c3cda2a5 (table test-table [id=b1fc3e6e4ee647b09711399040606c08]) successfully deleted
I20260808 15:53:36.531497 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 23996
I20260808 15:53:36.564152 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 24215
Aug 08 15:53:36 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:53:36.582706 24629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.565096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57452 (local address 127.27.84.194:40239)
0808 15:53:36.565324 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.565330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.565354 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:53:36.565512 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.565518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.565611 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.565745 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:36.567181 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.568480 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.569842 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.570502 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.570798 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.575303 (+ 4505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:36.575322 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:36.575344 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:36.575395 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:36.579568 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.580252 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.580262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.580266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.580461 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.580827 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.580833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.580836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.581432 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:36.581854 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.582028 (+ 174us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.582283 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:53:36.583572 24402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:36.702181 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 24215
I20260808 15:53:36.729350 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 24418
Aug 08 15:53:36 dist-test-slave-twwt krb5kdc[23838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:53:36.745615 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:36.730182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.27.84.195:40371)
0808 15:53:36.730374 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:36.730378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:36.730430 (+ 52us) server_negotiation.cc:407] Connection header received
0808 15:53:36.730586 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:36.730590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:36.730714 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:36.730803 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:36.732142 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.732901 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.734306 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:36.734929 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:36.735075 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:36.739710 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:36.739727 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:36.739744 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:36.739781 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:36.742591 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.743405 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.743415 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.743419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.743567 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:36.743978 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:36.743984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:36.743987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:36.744393 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:36.744700 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:36.745033 (+ 333us) server_negotiation.cc:299] Negotiation successful
0808 15:53:36.745268 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:53:36.746322 24602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:36.838650 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 24418
I20260808 15:53:36.865989 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 23855
I20260808 15:53:36.867274 23949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:36.980118 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 23855
2026-08-08T15:53:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4942 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-08T15:53:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:37Z Disabled control of system clock
I20260808 15:53:37.086036 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:45983
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:53:37.446180 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:37.446573 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:37.446764 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:37.454137 24689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:37.454254 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:37.454351 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:37.454464 24689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:37.454530 24689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:37.464104 24689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:45983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:37.465509 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:37.467679 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:37.480114 24696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:37.480157 24700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:37.480054 24694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:37.481343 24689 server_base.cc:1034] running on GCE node
I20260808 15:53:37.482209 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:37.483857 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:37.485148 24689 hybrid_clock.cc:648] HybridClock initialized: now 1786204417485092 us; error 49 us; skew 500 ppm
I20260808 15:53:37.485561 24689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:37.488171 24689 webserver.cc:460] Webserver started at http://127.27.84.254:41937/ using document root <none> and password file <none>
I20260808 15:53:37.488935 24689 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:37.489080 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:37.489450 24689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:37.492688 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "a6276b4d9f534023a90f8671a43352f0"
format_stamp: "Formatted at 2026-08-08 15:53:37 on dist-test-slave-twwt"
server_key: "2f3b742666aee472cab55b34b2a96f5e"
server_key_iv: "1b622434b0a8eb5b5960043890708b72"
server_key_version: "encryptionkey@0"
I20260808 15:53:37.493623 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "a6276b4d9f534023a90f8671a43352f0"
format_stamp: "Formatted at 2026-08-08 15:53:37 on dist-test-slave-twwt"
server_key: "2f3b742666aee472cab55b34b2a96f5e"
server_key_iv: "1b622434b0a8eb5b5960043890708b72"
server_key_version: "encryptionkey@0"
I20260808 15:53:37.499218 24689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:53:37.502869 24707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:37.504303 24689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:37.504508 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "a6276b4d9f534023a90f8671a43352f0"
format_stamp: "Formatted at 2026-08-08 15:53:37 on dist-test-slave-twwt"
server_key: "2f3b742666aee472cab55b34b2a96f5e"
server_key_iv: "1b622434b0a8eb5b5960043890708b72"
server_key_version: "encryptionkey@0"
I20260808 15:53:37.504733 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:37.524417 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:37.525309 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:37.525643 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:37.545868 24689 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:45983
I20260808 15:53:37.545895 24793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:45983 every 8 connection(s)
I20260808 15:53:37.547302 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:37.548236 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 24689
I20260808 15:53:37.548456 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:37.548909 27987 external_mini_cluster.cc:1428] Setting key 05115e0c4c84ce58e09f711e98834574
I20260808 15:53:37.551796 24794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:37.563239 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:37.550858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.27.84.254:45983)
0808 15:53:37.551892 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:37.551910 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:37.551958 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:53:37.553588 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:37.553609 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:37.554066 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:37.554422 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:37.555065 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:37.556452 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:37.558419 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:37.559158 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:37.559297 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:37.560580 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:37.560588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:37.560635 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:37.561268 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:37.561346 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:37.561374 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:37.561465 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:37.561718 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 15:53:37.562112 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":385,"threads_started":1}
I20260808 15:53:37.564718 24794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0: Bootstrap starting.
I20260808 15:53:37.568127 24794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:37.569228 24794 log.cc:824] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:37.572429 24794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0: No bootstrap required, opened a new log
I20260808 15:53:37.580039 24794 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6276b4d9f534023a90f8671a43352f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45983 } }
I20260808 15:53:37.580466 24794 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:37.580579 24794 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6276b4d9f534023a90f8671a43352f0, State: Initialized, Role: FOLLOWER
I20260808 15:53:37.581280 24794 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6276b4d9f534023a90f8671a43352f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45983 } }
I20260808 15:53:37.581497 24794 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:37.581662 24794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:37.581856 24794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:37.584239 24794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6276b4d9f534023a90f8671a43352f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45983 } }
I20260808 15:53:37.584789 24794 leader_election.cc:302] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6276b4d9f534023a90f8671a43352f0; no voters:
I20260808 15:53:37.585397 24794 leader_election.cc:288] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:37.585575 24800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:37.586371 24800 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [term 1 LEADER]: Becoming Leader. State: Replica: a6276b4d9f534023a90f8671a43352f0, State: Running, Role: LEADER
I20260808 15:53:37.586985 24800 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6276b4d9f534023a90f8671a43352f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45983 } }
I20260808 15:53:37.587671 24794 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:37.590018 24801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6276b4d9f534023a90f8671a43352f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6276b4d9f534023a90f8671a43352f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45983 } } }
I20260808 15:53:37.590453 24801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:37.590346 24803 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6276b4d9f534023a90f8671a43352f0. Latest consensus state: current_term: 1 leader_uuid: "a6276b4d9f534023a90f8671a43352f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6276b4d9f534023a90f8671a43352f0" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 45983 } } }
I20260808 15:53:37.591282 24803 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:37.594205 24808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:37.599808 24808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:37.611001 24808 catalog_manager.cc:1347] Generated new cluster ID: 4dc1ef5575ce47a992eb21c8a1b0c6d7
I20260808 15:53:37.611141 24808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:37.630580 24808 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:37.632324 24808 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:37.640987 24808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0: Generated new TSK 0
I20260808 15:53:37.641816 24808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:37.648931 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45983
--builtin_ntp_servers=127.27.84.212:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:53:38.030004 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:38.030352 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:38.030517 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:38.038335 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:38.038488 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:38.038731 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:38.048626 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:38.050045 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:38.052044 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:38.066094 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:38.066128 24833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:38.066246 24837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:38.066988 24824 server_base.cc:1034] running on GCE node
I20260808 15:53:38.067641 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:38.068665 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:38.069875 24824 hybrid_clock.cc:648] HybridClock initialized: now 1786204418069834 us; error 36 us; skew 500 ppm
I20260808 15:53:38.070206 24824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:38.072793 24824 webserver.cc:460] Webserver started at http://127.27.84.193:37293/ using document root <none> and password file <none>
I20260808 15:53:38.073568 24824 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:38.073665 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:38.074038 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:38.077052 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "02dd2df530974e5085e6a98267c1279b"
format_stamp: "Formatted at 2026-08-08 15:53:38 on dist-test-slave-twwt"
server_key: "19479e0d923f2e00431e5d2fb26aecb9"
server_key_iv: "e6db40c31aab9f9d39a8867c3f63673c"
server_key_version: "encryptionkey@0"
I20260808 15:53:38.077868 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "02dd2df530974e5085e6a98267c1279b"
format_stamp: "Formatted at 2026-08-08 15:53:38 on dist-test-slave-twwt"
server_key: "19479e0d923f2e00431e5d2fb26aecb9"
server_key_iv: "e6db40c31aab9f9d39a8867c3f63673c"
server_key_version: "encryptionkey@0"
I20260808 15:53:38.083184 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:53:38.086862 24847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:38.088447 24824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260808 15:53:38.088610 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "02dd2df530974e5085e6a98267c1279b"
format_stamp: "Formatted at 2026-08-08 15:53:38 on dist-test-slave-twwt"
server_key: "19479e0d923f2e00431e5d2fb26aecb9"
server_key_iv: "e6db40c31aab9f9d39a8867c3f63673c"
server_key_version: "encryptionkey@0"
I20260808 15:53:38.088829 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:38.125123 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:38.126067 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:38.126405 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:38.127511 24824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:38.129108 24824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:38.129231 24824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:38.129374 24824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:38.129457 24824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:38.166867 24824 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:41181
I20260808 15:53:38.166994 25016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:41181 every 8 connection(s)
I20260808 15:53:38.168256 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:38.170090 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 24824
I20260808 15:53:38.170293 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:53:38.170881 27987 external_mini_cluster.cc:1428] Setting key 336db427b815042a693477059840c693
I20260808 15:53:38.174336 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45983
--builtin_ntp_servers=127.27.84.212:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:53:38.184186 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:38.171317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:55117 (local address 127.27.84.254:45983)
0808 15:53:38.171872 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:38.171878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:38.173302 (+ 1424us) server_negotiation.cc:407] Connection header received
0808 15:53:38.176127 (+ 2825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:38.176131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:38.176195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:38.176292 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:38.177732 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.178458 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.181008 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.181430 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.181542 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:38.183374 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:38.183379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:38.183382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:38.183401 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:38.183442 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:38.183455 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:38.183526 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:38.183813 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:53:38.183921 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":308,"threads_started":1}
I20260808 15:53:38.185702 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:38.171939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:45983 (local address 127.27.84.193:55117)
0808 15:53:38.173101 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:38.173182 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:38.175800 (+ 2618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:38.176572 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:38.176586 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:38.176709 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:38.177538 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:38.177565 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.178655 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.178664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:38.180833 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:38.180851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.181680 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.181690 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:38.182045 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:38.182955 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:38.182961 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:38.183017 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:38.183135 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:38.183560 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:38.183575 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:38.183586 (+ 11us) client_negotiation.cc:769] Sending connection context
0808 15:53:38.183785 (+ 199us) client_negotiation.cc:240] Negotiation successful
0808 15:53:38.184235 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":340,"threads_started":1}
I20260808 15:53:38.187626 25017 heartbeater.cc:347] Connected to a master server at 127.27.84.254:45983
I20260808 15:53:38.187983 25017 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:38.188992 25017 heartbeater.cc:511] Master 127.27.84.254:45983 requested a full tablet report, sending...
I20260808 15:53:38.191239 24739 ts_manager.cc:194] Registered new tserver with Master: 02dd2df530974e5085e6a98267c1279b (127.27.84.193:41181)
I20260808 15:53:38.193010 24739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.193:55117
W20260808 15:53:38.532742 25022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:38.533034 25022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:38.533123 25022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:38.540307 25022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:38.540403 25022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:38.540509 25022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:38.549468 25022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:38.550761 25022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:38.552552 25022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:38.566531 25034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:38.566524 25030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:38.566679 25029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:38.567314 25022 server_base.cc:1034] running on GCE node
I20260808 15:53:38.568037 25022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:38.569012 25022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:38.570247 25022 hybrid_clock.cc:648] HybridClock initialized: now 1786204418570199 us; error 60 us; skew 500 ppm
I20260808 15:53:38.570681 25022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:38.573354 25022 webserver.cc:460] Webserver started at http://127.27.84.194:46535/ using document root <none> and password file <none>
I20260808 15:53:38.574163 25022 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:38.574326 25022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:38.574766 25022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:38.578043 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "41d59a5a141f46dab8ecb8d5ff506e14"
format_stamp: "Formatted at 2026-08-08 15:53:38 on dist-test-slave-twwt"
server_key: "bc845543ecad027a5f03181cd5b10c90"
server_key_iv: "2ea6eb76de7ffc277d999897ab6eb810"
server_key_version: "encryptionkey@0"
I20260808 15:53:38.579025 25022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "41d59a5a141f46dab8ecb8d5ff506e14"
format_stamp: "Formatted at 2026-08-08 15:53:38 on dist-test-slave-twwt"
server_key: "bc845543ecad027a5f03181cd5b10c90"
server_key_iv: "2ea6eb76de7ffc277d999897ab6eb810"
server_key_version: "encryptionkey@0"
I20260808 15:53:38.584717 25022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260808 15:53:38.588976 25042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:38.590396 25022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:53:38.590598 25022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "41d59a5a141f46dab8ecb8d5ff506e14"
format_stamp: "Formatted at 2026-08-08 15:53:38 on dist-test-slave-twwt"
server_key: "bc845543ecad027a5f03181cd5b10c90"
server_key_iv: "2ea6eb76de7ffc277d999897ab6eb810"
server_key_version: "encryptionkey@0"
I20260808 15:53:38.590929 25022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:38.618911 25022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:38.619830 25022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:38.620143 25022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:38.621281 25022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:38.623009 25022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:38.623108 25022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:38.623196 25022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:38.623261 25022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:38.660760 25022 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:45495
I20260808 15:53:38.660914 25204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:45495 every 8 connection(s)
I20260808 15:53:38.662221 25022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:38.672286 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 25022
I20260808 15:53:38.672510 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:53:38.673031 27987 external_mini_cluster.cc:1428] Setting key 96ae7f69c687285075293236ff9b26ba
I20260808 15:53:38.676872 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:38.665303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:43971 (local address 127.27.84.254:45983)
0808 15:53:38.665548 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:38.665555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:38.667252 (+ 1697us) server_negotiation.cc:407] Connection header received
0808 15:53:38.669183 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:38.669188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:38.669259 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:38.669369 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:38.670504 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.671395 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.673299 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.673924 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.674095 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:38.675878 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:38.675885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:38.675890 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:38.675913 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:38.675953 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:38.675973 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:38.676077 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:38.676322 (+ 245us) server_negotiation.cc:299] Negotiation successful
0808 15:53:38.676447 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260808 15:53:38.676903 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45983
--builtin_ntp_servers=127.27.84.212:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:53:38.677953 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:38.665966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:45983 (local address 127.27.84.194:43971)
0808 15:53:38.667072 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:38.667113 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:38.668918 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:38.669581 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:38.669590 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:38.669687 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:38.670338 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:38.670357 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.671555 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.671560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:38.673102 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:38.673116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:38.674194 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:38.674199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:38.674550 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:38.675472 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:38.675476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:38.675537 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:38.675663 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:38.676086 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:38.676096 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:38.676103 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:53:38.676285 (+ 182us) client_negotiation.cc:240] Negotiation successful
0808 15:53:38.676660 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":328,"threads_started":1}
I20260808 15:53:38.679472 25206 heartbeater.cc:347] Connected to a master server at 127.27.84.254:45983
I20260808 15:53:38.679831 25206 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:38.680801 25206 heartbeater.cc:511] Master 127.27.84.254:45983 requested a full tablet report, sending...
I20260808 15:53:38.682438 24739 ts_manager.cc:194] Registered new tserver with Master: 41d59a5a141f46dab8ecb8d5ff506e14 (127.27.84.194:45495)
I20260808 15:53:38.683439 24739 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.194:43971
W20260808 15:53:39.043469 25209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:39.043825 25209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:39.044006 25209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:39.051421 25209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:39.051559 25209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:39.051750 25209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:39.061072 25209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:45983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:39.062534 25209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:39.064749 25209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:39.079396 25215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:39.079499 25222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:39.079452 25216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:39.080753 25209 server_base.cc:1034] running on GCE node
I20260808 15:53:39.081411 25209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:39.082442 25209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:39.083675 25209 hybrid_clock.cc:648] HybridClock initialized: now 1786204419083626 us; error 68 us; skew 500 ppm
I20260808 15:53:39.084025 25209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:39.086409 25209 webserver.cc:460] Webserver started at http://127.27.84.195:44191/ using document root <none> and password file <none>
I20260808 15:53:39.087210 25209 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:39.087347 25209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:39.087690 25209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:39.090835 25209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "8aea7d375eab4b5ea6b46fc17d0122ec"
format_stamp: "Formatted at 2026-08-08 15:53:39 on dist-test-slave-twwt"
server_key: "052456cc2891c133a7b41228b9ad8f2c"
server_key_iv: "2d9d984a2cc15e49643b20599772599a"
server_key_version: "encryptionkey@0"
I20260808 15:53:39.091619 25209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "8aea7d375eab4b5ea6b46fc17d0122ec"
format_stamp: "Formatted at 2026-08-08 15:53:39 on dist-test-slave-twwt"
server_key: "052456cc2891c133a7b41228b9ad8f2c"
server_key_iv: "2d9d984a2cc15e49643b20599772599a"
server_key_version: "encryptionkey@0"
I20260808 15:53:39.097477 25209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:53:39.101123 25233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:39.102509 25209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:53:39.102738 25209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "8aea7d375eab4b5ea6b46fc17d0122ec"
format_stamp: "Formatted at 2026-08-08 15:53:39 on dist-test-slave-twwt"
server_key: "052456cc2891c133a7b41228b9ad8f2c"
server_key_iv: "2d9d984a2cc15e49643b20599772599a"
server_key_version: "encryptionkey@0"
I20260808 15:53:39.102955 25209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:39.125288 25209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:39.126221 25209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:39.126565 25209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:39.127681 25209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:39.129309 25209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:39.129405 25209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:39.129570 25209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:39.129657 25209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:39.166491 25209 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:37571
I20260808 15:53:39.166491 25402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:37571 every 8 connection(s)
I20260808 15:53:39.167893 25209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:39.174458 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 25209
I20260808 15:53:39.174688 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:53:39.175294 27987 external_mini_cluster.cc:1428] Setting key 2f0e7ce602bbeb198d9e38029387a506
I20260808 15:53:39.188055 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.170967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:54063 (local address 127.27.84.254:45983)
0808 15:53:39.171195 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.171202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.172629 (+ 1427us) server_negotiation.cc:407] Connection header received
0808 15:53:39.174684 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.174695 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.174808 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.174897 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:39.176248 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.177160 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.178950 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.179343 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.179483 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.187122 (+ 7639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:39.187128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:39.187131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:39.187151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:39.187185 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:39.187199 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:39.187324 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.187591 (+ 267us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.187706 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260808 15:53:39.189357 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.171519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:45983 (local address 127.27.84.195:54063)
0808 15:53:39.172472 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.172508 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:39.174330 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:39.175178 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:39.175189 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:39.175310 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:39.176069 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.176091 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.177323 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.177327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.178803 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.178817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.179591 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.179599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.179829 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.186676 (+ 6847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:39.186683 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:39.186741 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:39.186878 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:39.187309 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:39.187322 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:39.187331 (+ 9us) client_negotiation.cc:769] Sending connection context
0808 15:53:39.187541 (+ 210us) client_negotiation.cc:240] Negotiation successful
0808 15:53:39.188027 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":305,"threads_started":1}
I20260808 15:53:39.191027 25403 heartbeater.cc:347] Connected to a master server at 127.27.84.254:45983
I20260808 15:53:39.191347 25403 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:39.192292 25403 heartbeater.cc:511] Master 127.27.84.254:45983 requested a full tablet report, sending...
I20260808 15:53:39.193974 24738 ts_manager.cc:194] Registered new tserver with Master: 8aea7d375eab4b5ea6b46fc17d0122ec (127.27.84.195:37571)
I20260808 15:53:39.194989 24738 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.195:54063
I20260808 15:53:39.196581 25017 heartbeater.cc:503] Master 127.27.84.254:45983 was elected leader, sending a full tablet report...
I20260808 15:53:39.203893 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:53:39.215305 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.208962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.27.84.254:45983)
0808 15:53:39.209248 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.209255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.209568 (+ 313us) server_negotiation.cc:407] Connection header received
0808 15:53:39.209766 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.209771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.209854 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.209960 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:39.210467 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.211241 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.212693 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.213144 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.213288 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.214477 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:39.214483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:39.214495 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:39.214513 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:39.214545 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:39.214557 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:39.214733 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.214973 (+ 240us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.215028 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:53:39.219725 24738 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:53:39.226850 24738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:53:39.263079 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.239900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:37571 (local address 127.0.0.1:57526)
0808 15:53:39.253640 (+ 13740us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.253697 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:39.253924 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:39.255177 (+ 1253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:39.255189 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:39.255223 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:39.255681 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.255693 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.257927 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.257936 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.260427 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.260445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.261944 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.261950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.262186 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.262251 (+ 65us) client_negotiation.cc:769] Sending connection context
0808 15:53:39.262426 (+ 175us) client_negotiation.cc:240] Negotiation successful
0808 15:53:39.262665 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13460,"mutex_wait_us":7,"thread_start_us":13289,"threads_started":1}
I20260808 15:53:39.263957 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.240022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.27.84.195:37571)
0808 15:53:39.253157 (+ 13135us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.253166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.253966 (+ 800us) server_negotiation.cc:407] Connection header received
0808 15:53:39.254270 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.254279 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.254587 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.254823 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:39.256066 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.257782 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.260658 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.261727 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.261912 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.262240 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.263371 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.263566 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12947,"thread_start_us":12794,"threads_started":1}
I20260808 15:53:39.264907 25427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.239914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:45495 (local address 127.0.0.1:41810)
0808 15:53:39.254436 (+ 14522us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.254460 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:39.254702 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:39.255673 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:39.255677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:39.255700 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:39.256350 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.256372 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.258546 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.258551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.260724 (+ 2173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.260741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.264078 (+ 3337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.264087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.264248 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.264287 (+ 39us) client_negotiation.cc:769] Sending connection context
0808 15:53:39.264401 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:53:39.264535 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14400,"mutex_wait_us":15,"thread_start_us":16375,"threads_started":1}
I20260808 15:53:39.265457 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.242945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50384)
0808 15:53:39.256618 (+ 13673us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.256645 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:39.256852 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:39.257510 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:39.257515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:39.257541 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:39.257900 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.257912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.260058 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.260064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.262217 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.262236 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.264631 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.264637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.264801 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.264850 (+ 49us) client_negotiation.cc:769] Sending connection context
0808 15:53:39.264950 (+ 100us) client_negotiation.cc:240] Negotiation successful
0808 15:53:39.265081 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13527,"mutex_wait_us":4,"thread_start_us":11217,"threads_started":1}
I20260808 15:53:39.265877 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.243252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.27.84.193:41181)
0808 15:53:39.255932 (+ 12680us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.255944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.256777 (+ 833us) server_negotiation.cc:407] Connection header received
0808 15:53:39.257090 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.257099 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.257306 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.257503 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:39.258118 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.259895 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.262454 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.264034 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.264184 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.264498 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.265287 (+ 789us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.265481 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12488,"thread_start_us":12336,"threads_started":1}
I20260808 15:53:39.266979 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.240216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.27.84.194:45495)
0808 15:53:39.251063 (+ 10847us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.251078 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.254567 (+ 3489us) server_negotiation.cc:407] Connection header received
0808 15:53:39.254914 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.254921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.255157 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.255342 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:39.256624 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.258363 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.260948 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.262912 (+ 1964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.263049 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.263942 (+ 893us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.264459 (+ 517us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.264834 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10595,"thread_start_us":10421,"threads_started":1}
I20260808 15:53:39.268393 25303 tablet_service.cc:1467] Processing CreateTablet for tablet 1782a642dce741959075adc698666080 (DEFAULT_TABLE table=test-table [id=cd3222e0b9a64b95b547b55c90b5758c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:39.270170 25303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1782a642dce741959075adc698666080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:39.269703 24928 tablet_service.cc:1467] Processing CreateTablet for tablet 1782a642dce741959075adc698666080 (DEFAULT_TABLE table=test-table [id=cd3222e0b9a64b95b547b55c90b5758c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:39.269889 25108 tablet_service.cc:1467] Processing CreateTablet for tablet 1782a642dce741959075adc698666080 (DEFAULT_TABLE table=test-table [id=cd3222e0b9a64b95b547b55c90b5758c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:39.271235 25108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1782a642dce741959075adc698666080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:39.271265 24928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1782a642dce741959075adc698666080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:39.283739 25430 tablet_bootstrap.cc:492] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Bootstrap starting.
I20260808 15:53:39.288102 25432 tablet_bootstrap.cc:492] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Bootstrap starting.
I20260808 15:53:39.288501 25431 tablet_bootstrap.cc:492] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: Bootstrap starting.
I20260808 15:53:39.289412 25430 tablet_bootstrap.cc:654] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:39.290875 25430 log.cc:824] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:39.292281 25431 tablet_bootstrap.cc:654] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:39.293608 25431 log.cc:824] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:39.294659 25432 tablet_bootstrap.cc:654] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:39.295854 25430 tablet_bootstrap.cc:492] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: No bootstrap required, opened a new log
I20260808 15:53:39.296078 25432 log.cc:824] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:39.296263 25430 ts_tablet_manager.cc:1397] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20260808 15:53:39.297144 25431 tablet_bootstrap.cc:492] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: No bootstrap required, opened a new log
I20260808 15:53:39.297442 25431 ts_tablet_manager.cc:1397] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260808 15:53:39.299021 25432 tablet_bootstrap.cc:492] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: No bootstrap required, opened a new log
I20260808 15:53:39.299301 25432 ts_tablet_manager.cc:1397] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260808 15:53:39.307406 25432 raft_consensus.cc:348] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.307420 25430 raft_consensus.cc:348] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.307835 25432 raft_consensus.cc:374] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:39.307858 25430 raft_consensus.cc:374] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:39.307942 25432 raft_consensus.cc:729] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8aea7d375eab4b5ea6b46fc17d0122ec, State: Initialized, Role: FOLLOWER
I20260808 15:53:39.307964 25430 raft_consensus.cc:729] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d59a5a141f46dab8ecb8d5ff506e14, State: Initialized, Role: FOLLOWER
I20260808 15:53:39.308393 25431 raft_consensus.cc:348] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.308835 25431 raft_consensus.cc:374] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:39.308946 25431 raft_consensus.cc:729] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dd2df530974e5085e6a98267c1279b, State: Initialized, Role: FOLLOWER
I20260808 15:53:39.308779 25430 consensus_queue.cc:260] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.308939 25432 consensus_queue.cc:260] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.309389 25437 raft_consensus.cc:482] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:39.309810 25431 consensus_queue.cc:260] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.309756 25437 raft_consensus.cc:504] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.317911 25432 ts_tablet_manager.cc:1428] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260808 15:53:39.318341 25403 heartbeater.cc:503] Master 127.27.84.254:45983 was elected leader, sending a full tablet report...
I20260808 15:53:39.318429 25431 ts_tablet_manager.cc:1428] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: Time spent starting tablet: real 0.021s user 0.021s sys 0.000s
I20260808 15:53:39.320959 25437 leader_election.cc:288] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41d59a5a141f46dab8ecb8d5ff506e14 (127.27.84.194:45495), 02dd2df530974e5085e6a98267c1279b (127.27.84.193:41181)
I20260808 15:53:39.322052 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.312626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:45495 (local address 127.27.84.195:60157)
0808 15:53:39.313172 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.313208 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:39.313367 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:39.314172 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:39.314180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:39.314232 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:39.314524 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.314539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.316523 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.316532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.318794 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.318813 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.320551 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.320557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.320761 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.320934 (+ 173us) client_negotiation.cc:769] Sending connection context
0808 15:53:39.321591 (+ 657us) client_negotiation.cc:240] Negotiation successful
0808 15:53:39.321763 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357}
I20260808 15:53:39.322537 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.312894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:60157 (local address 127.27.84.194:45495)
0808 15:53:39.313617 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.313623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.313643 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:53:39.313729 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.313737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.313851 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.313962 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:39.314809 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.316265 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.319009 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.320174 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.320349 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.320835 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.321184 (+ 349us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.321387 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590}
I20260808 15:53:39.323735 25141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1782a642dce741959075adc698666080" candidate_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" is_pre_election: true
I20260808 15:53:39.324580 25141 raft_consensus.cc:2454] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8aea7d375eab4b5ea6b46fc17d0122ec in term 0.
I20260808 15:53:39.325762 25236 leader_election.cc:302] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d59a5a141f46dab8ecb8d5ff506e14, 8aea7d375eab4b5ea6b46fc17d0122ec; no voters:
I20260808 15:53:39.326789 25439 raft_consensus.cc:2781] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:53:39.327075 25439 raft_consensus.cc:482] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:53:39.327317 25439 raft_consensus.cc:3037] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:39.329764 25206 heartbeater.cc:503] Master 127.27.84.254:45983 was elected leader, sending a full tablet report...
I20260808 15:53:39.331027 25430 ts_tablet_manager.cc:1428] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Time spent starting tablet: real 0.035s user 0.009s sys 0.011s
I20260808 15:53:39.331748 25439 raft_consensus.cc:504] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.332877 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.313418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:54591 (local address 127.27.84.193:41181)
0808 15:53:39.313613 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.313619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.323411 (+ 9792us) server_negotiation.cc:407] Connection header received
0808 15:53:39.323687 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.323693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.323804 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.323875 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:39.326194 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.327634 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.330349 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.331688 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.331863 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.331947 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.332448 (+ 501us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.332623 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:53:39.332901 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.313536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.27.84.195:54591)
0808 15:53:39.323305 (+ 9769us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.323328 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:39.323522 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:39.325768 (+ 2246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:39.325780 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:39.325816 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:39.326049 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.326065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.327876 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.327885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.330128 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:39.330144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.331985 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.331992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:39.332269 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.332295 (+ 26us) client_negotiation.cc:769] Sending connection context
0808 15:53:39.332404 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:53:39.332542 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9628,"thread_start_us":18143,"threads_started":1}
I20260808 15:53:39.333395 25141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1782a642dce741959075adc698666080" candidate_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d59a5a141f46dab8ecb8d5ff506e14"
I20260808 15:53:39.333623 25141 raft_consensus.cc:3037] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:39.333670 25439 leader_election.cc:288] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [CANDIDATE]: Term 1 election: Requested vote from peers 41d59a5a141f46dab8ecb8d5ff506e14 (127.27.84.194:45495), 02dd2df530974e5085e6a98267c1279b (127.27.84.193:41181)
I20260808 15:53:39.334136 24958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1782a642dce741959075adc698666080" candidate_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dd2df530974e5085e6a98267c1279b" is_pre_election: true
I20260808 15:53:39.334472 24956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1782a642dce741959075adc698666080" candidate_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02dd2df530974e5085e6a98267c1279b"
I20260808 15:53:39.334899 24956 raft_consensus.cc:3037] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:39.336961 25141 raft_consensus.cc:2454] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aea7d375eab4b5ea6b46fc17d0122ec in term 1.
I20260808 15:53:39.337419 25236 leader_election.cc:302] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d59a5a141f46dab8ecb8d5ff506e14, 8aea7d375eab4b5ea6b46fc17d0122ec; no voters:
I20260808 15:53:39.337690 25439 raft_consensus.cc:2781] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:39.337896 25439 raft_consensus.cc:686] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 1 LEADER]: Becoming Leader. State: Replica: 8aea7d375eab4b5ea6b46fc17d0122ec, State: Running, Role: LEADER
I20260808 15:53:39.338116 24956 raft_consensus.cc:2454] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8aea7d375eab4b5ea6b46fc17d0122ec in term 1.
I20260808 15:53:39.338411 25439 consensus_queue.cc:237] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } }
I20260808 15:53:39.343226 24739 catalog_manager.cc:5495] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 8aea7d375eab4b5ea6b46fc17d0122ec (127.27.84.195). New cstate: current_term: 1 leader_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8aea7d375eab4b5ea6b46fc17d0122ec" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 37571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 } health_report { overall_health: UNKNOWN } } }
I20260808 15:53:39.369452 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.363416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57540 (local address 127.27.84.195:37571)
0808 15:53:39.363619 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.363624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.363659 (+ 35us) server_negotiation.cc:407] Connection header received
0808 15:53:39.363747 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.363751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.363842 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.363973 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:53:39.364399 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.365354 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.366841 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.367352 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.367473 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.368100 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:53:39.368675 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:53:39.368842 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.369110 (+ 268us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.369165 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:53:39.377826 24956 raft_consensus.cc:1261] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 1 FOLLOWER]: Refusing update from remote peer 8aea7d375eab4b5ea6b46fc17d0122ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:39.377916 25141 raft_consensus.cc:1261] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 1 FOLLOWER]: Refusing update from remote peer 8aea7d375eab4b5ea6b46fc17d0122ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:39.378836 25437 consensus_queue.cc:1035] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "02dd2df530974e5085e6a98267c1279b" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:39.379156 25439 consensus_queue.cc:1035] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d59a5a141f46dab8ecb8d5ff506e14" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 45495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:39.407615 24738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:39.407985 24738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:39.411973 24738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6276b4d9f534023a90f8671a43352f0: Sending DeleteTablet for 3 replicas of tablet 1782a642dce741959075adc698666080
I20260808 15:53:39.413285 25303 tablet_service.cc:1514] Processing DeleteTablet for tablet 1782a642dce741959075adc698666080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:39 UTC) from {username='slave'} at 127.0.0.1:57526
I20260808 15:53:39.413434 24928 tablet_service.cc:1514] Processing DeleteTablet for tablet 1782a642dce741959075adc698666080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:39 UTC) from {username='slave'} at 127.0.0.1:50384
I20260808 15:53:39.413468 25108 tablet_service.cc:1514] Processing DeleteTablet for tablet 1782a642dce741959075adc698666080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:39 UTC) from {username='slave'} at 127.0.0.1:41810
I20260808 15:53:39.415681 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 24824
I20260808 15:53:39.426524 25465 tablet_replica.cc:330] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: stopping tablet replica
I20260808 15:53:39.427248 25463 tablet_replica.cc:330] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: stopping tablet replica
I20260808 15:53:39.427693 25463 raft_consensus.cc:2229] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:39.428207 25463 raft_consensus.cc:2258] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:39.429582 25465 raft_consensus.cc:2229] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:39.430083 25465 raft_consensus.cc:2258] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:39.432639 25465 ts_tablet_manager.cc:1905] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:39.433939 25463 ts_tablet_manager.cc:1905] T 1782a642dce741959075adc698666080 P 02dd2df530974e5085e6a98267c1279b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:39.434967 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.417117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.27.84.193:41181)
0808 15:53:39.417322 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.417328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.421889 (+ 4561us) server_negotiation.cc:407] Connection header received
0808 15:53:39.422122 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.422128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.422263 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.422397 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:39.426795 (+ 4398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.428090 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.430111 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.430936 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.431103 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.432805 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:39.432820 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:39.432842 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:39.433640 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:39.433699 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:39.433725 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:39.433806 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.434437 (+ 631us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.434563 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260808 15:53:39.435698 25011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:39.440586 25462 tablet_replica.cc:330] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: stopping tablet replica
I20260808 15:53:39.441012 25462 raft_consensus.cc:2229] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:53:39.441476 25465 ts_tablet_manager.cc:1918] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:39.441651 25465 log.cc:1196] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/1782a642dce741959075adc698666080
I20260808 15:53:39.441655 25462 raft_consensus.cc:2258] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:39.442255 25465 ts_tablet_manager.cc:1939] T 1782a642dce741959075adc698666080 P 41d59a5a141f46dab8ecb8d5ff506e14: Deleting consensus metadata
I20260808 15:53:39.444236 25462 ts_tablet_manager.cc:1905] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:39.444321 24710 catalog_manager.cc:4841] TS 41d59a5a141f46dab8ecb8d5ff506e14 (127.27.84.194:45495): tablet 1782a642dce741959075adc698666080 (table test-table [id=cd3222e0b9a64b95b547b55c90b5758c]) successfully deleted
I20260808 15:53:39.452823 25462 ts_tablet_manager.cc:1918] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:39.453114 25462 log.cc:1196] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/1782a642dce741959075adc698666080
I20260808 15:53:39.453863 25462 ts_tablet_manager.cc:1939] T 1782a642dce741959075adc698666080 P 8aea7d375eab4b5ea6b46fc17d0122ec: Deleting consensus metadata
I20260808 15:53:39.456291 24709 catalog_manager.cc:4841] TS 8aea7d375eab4b5ea6b46fc17d0122ec (127.27.84.195:37571): tablet 1782a642dce741959075adc698666080 (table test-table [id=cd3222e0b9a64b95b547b55c90b5758c]) successfully deleted
I20260808 15:53:39.533571 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 24824
W20260808 15:53:39.566094 24713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:41181 (error 108)
I20260808 15:53:39.570796 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 25022
I20260808 15:53:39.571995 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.571513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50396)
0808 15:53:39.571690 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.571812 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:53:39.572423 24713 catalog_manager.cc:4584] TS 02dd2df530974e5085e6a98267c1279b (127.27.84.193:41181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1782a642dce741959075adc698666080: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
I20260808 15:53:39.585289 25426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.575386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41818 (local address 127.27.84.194:45495)
0808 15:53:39.575602 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.575610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.575632 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:53:39.575706 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.575714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.576681 (+ 967us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.576796 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:39.577357 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.578759 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.580730 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.581588 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.581750 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.583225 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:39.583239 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:39.583261 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:39.584242 (+ 981us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:39.584308 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:39.584338 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:39.584465 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.584640 (+ 175us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.584782 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:53:39.586052 25197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:39.626636 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.625968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50412)
0808 15:53:39.626251 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.626398 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260808 15:53:39.627887 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.627458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50428)
0808 15:53:39.627632 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.627719 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260808 15:53:39.679775 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 25022
I20260808 15:53:39.688597 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.688016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50442)
0808 15:53:39.688230 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.688359 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:53:39.689898 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.689540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50448)
0808 15:53:39.689671 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.689753 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260808 15:53:39.706172 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 25209
I20260808 15:53:39.714097 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.707339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.27.84.195:37571)
0808 15:53:39.707557 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:39.707563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:39.707588 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:53:39.707644 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:39.707648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:39.707724 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:39.707850 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:39.708339 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.709251 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.710507 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:39.711200 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:39.711357 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:39.712510 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:39.712520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:39.712537 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:39.713223 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:39.713282 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:39.713303 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:39.713445 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:39.713585 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 15:53:39.713690 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:53:39.714864 25396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:39.770772 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.770205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50454)
0808 15:53:39.770431 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.770532 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 15:53:39.772028 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.771577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50458)
0808 15:53:39.771807 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.771913 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260808 15:53:39.850833 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 25209
I20260808 15:53:39.881752 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 24689
I20260808 15:53:39.883067 24788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:39.997857 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 24689
I20260808 15:53:39.997913 25421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:39.997169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41181 (local address 127.0.0.1:50462)
0808 15:53:39.997503 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:39.997671 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
2026-08-08T15:53:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:40 dist-test-slave-twwt krb5kdc[25480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:40 dist-test-slave-twwt krb5kdc[25480](info): set up 2 sockets
Aug 08 15:53:40 dist-test-slave-twwt krb5kdc[25480](info): commencing operation
krb5kdc: starting...
W20260808 15:53:41.444099 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.395s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:41 dist-test-slave-twwt krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:53:41.795998 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41401
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:41401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:42.162886 25496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:42.163192 25496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:42.163350 25496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:42.170841 25496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:42.170961 25496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:42.171046 25496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:42.171145 25496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:42.171231 25496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:42.181563 25496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:41401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:42.182987 25496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:42.184389 25496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:42.196910 25506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:42.196995 25507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:42.197192 25509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:42.198036 25496 server_base.cc:1034] running on GCE node
I20260808 15:53:42.198994 25496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:42.200522 25496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:42.201817 25496 hybrid_clock.cc:648] HybridClock initialized: now 1786204422201746 us; error 57 us; skew 500 ppm
I20260808 15:53:42.202219 25496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:42 dist-test-slave-twwt krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:42.205718 25496 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:53:42.207499 25496 webserver.cc:460] Webserver started at http://127.27.84.254:39063/ using document root <none> and password file <none>
I20260808 15:53:42.208209 25496 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:42.208313 25496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:42.208557 25496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:42.211788 25496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "fde28bd0e8c946aa98b3073464e15d43"
format_stamp: "Formatted at 2026-08-08 15:53:42 on dist-test-slave-twwt"
server_key: "84c38539cac7667318d09f8db5c6a8ae"
server_key_iv: "082656148aab897b3c118839f9dea524"
server_key_version: "encryptionkey@0"
I20260808 15:53:42.212522 25496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "fde28bd0e8c946aa98b3073464e15d43"
format_stamp: "Formatted at 2026-08-08 15:53:42 on dist-test-slave-twwt"
server_key: "84c38539cac7667318d09f8db5c6a8ae"
server_key_iv: "082656148aab897b3c118839f9dea524"
server_key_version: "encryptionkey@0"
I20260808 15:53:42.218070 25496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260808 15:53:42.221974 25521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:42.223855 25496 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260808 15:53:42.224043 25496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "fde28bd0e8c946aa98b3073464e15d43"
format_stamp: "Formatted at 2026-08-08 15:53:42 on dist-test-slave-twwt"
server_key: "84c38539cac7667318d09f8db5c6a8ae"
server_key_iv: "082656148aab897b3c118839f9dea524"
server_key_version: "encryptionkey@0"
I20260808 15:53:42.224239 25496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:42.240221 25496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:42.244692 25496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:42.244967 25496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:42.264336 25496 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:41401
I20260808 15:53:42.264385 25604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:41401 every 8 connection(s)
I20260808 15:53:42.265802 25496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:42.269778 25605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:42.270918 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 25496
I20260808 15:53:42.271106 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:42.271580 27987 external_mini_cluster.cc:1428] Setting key aee9af13e0ed4c5932fab5a79fec8284
I20260808 15:53:42.284837 25605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43: Bootstrap starting.
Aug 08 15:53:42 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:42.288694 25605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:42.289820 25605 log.cc:824] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:42.292846 25605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43: No bootstrap required, opened a new log
I20260808 15:53:42.296571 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:42.273664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.27.84.254:41401)
0808 15:53:42.274576 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:42.274589 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:42.274693 (+ 104us) server_negotiation.cc:407] Connection header received
0808 15:53:42.276410 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:42.276434 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:42.276850 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:42.277205 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:42.279206 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:42.280597 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:42.282561 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:42.283342 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:42.283487 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:42.288511 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:42.288519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:42.288534 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:42.288572 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:42.292043 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:42.292884 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:42.292892 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:42.292899 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:42.293129 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:42.293581 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:42.293587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:42.293590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:42.294154 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:42.294522 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:42.295066 (+ 544us) server_negotiation.cc:299] Negotiation successful
0808 15:53:42.295540 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":408,"threads_started":1}
I20260808 15:53:42.301771 25605 raft_consensus.cc:348] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde28bd0e8c946aa98b3073464e15d43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41401 } }
I20260808 15:53:42.302282 25605 raft_consensus.cc:374] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:42.302424 25605 raft_consensus.cc:729] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde28bd0e8c946aa98b3073464e15d43, State: Initialized, Role: FOLLOWER
I20260808 15:53:42.303038 25605 consensus_queue.cc:260] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde28bd0e8c946aa98b3073464e15d43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41401 } }
I20260808 15:53:42.303297 25605 raft_consensus.cc:388] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:42.303458 25605 raft_consensus.cc:482] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:42.303653 25605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:42.305848 25605 raft_consensus.cc:504] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde28bd0e8c946aa98b3073464e15d43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41401 } }
I20260808 15:53:42.306365 25605 leader_election.cc:302] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fde28bd0e8c946aa98b3073464e15d43; no voters:
I20260808 15:53:42.307070 25605 leader_election.cc:288] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:42.307222 25614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:42.308084 25614 raft_consensus.cc:686] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [term 1 LEADER]: Becoming Leader. State: Replica: fde28bd0e8c946aa98b3073464e15d43, State: Running, Role: LEADER
I20260808 15:53:42.308768 25614 consensus_queue.cc:237] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde28bd0e8c946aa98b3073464e15d43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41401 } }
I20260808 15:53:42.309577 25605 sys_catalog.cc:564] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:42.311749 25618 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fde28bd0e8c946aa98b3073464e15d43. Latest consensus state: current_term: 1 leader_uuid: "fde28bd0e8c946aa98b3073464e15d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde28bd0e8c946aa98b3073464e15d43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41401 } } }
I20260808 15:53:42.312098 25618 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:42.313458 25615 sys_catalog.cc:455] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fde28bd0e8c946aa98b3073464e15d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fde28bd0e8c946aa98b3073464e15d43" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 41401 } } }
I20260808 15:53:42.314621 25615 sys_catalog.cc:458] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:42.314751 25629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:42.320493 25629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:42.330209 25629 catalog_manager.cc:1347] Generated new cluster ID: 9165c5c1d3cf45af9e87f3bb4cbcb26b
I20260808 15:53:42.330366 25629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:42.365247 25629 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:42.366767 25629 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:42.380831 25629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43: Generated new TSK 0
I20260808 15:53:42.381695 25629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.84.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
Entry for principal kudu/127.27.84.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab.
I20260808 15:53:42.476655 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41401
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:42.841753 25656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:42.842087 25656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:42.842264 25656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:42.850908 25656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:42.851059 25656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:42.851249 25656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:42.861214 25656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.193.keytab
--principal=kudu/127.27.84.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:42.862597 25656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:42.864111 25656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:42.878335 25662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:42.878364 25663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:42.878410 25667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:42.879324 25656 server_base.cc:1034] running on GCE node
I20260808 15:53:42.880009 25656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:42.881163 25656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:42.882467 25656 hybrid_clock.cc:648] HybridClock initialized: now 1786204422882375 us; error 61 us; skew 500 ppm
I20260808 15:53:42.882890 25656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:42 dist-test-slave-twwt krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:42.886348 25656 init.cc:377] Logged in from keytab as kudu/127.27.84.193@KRBTEST.COM (short username kudu)
I20260808 15:53:42.888093 25656 webserver.cc:460] Webserver started at http://127.27.84.193:40557/ using document root <none> and password file <none>
I20260808 15:53:42.888854 25656 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:42.889019 25656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:42.889376 25656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:42.892676 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "8f1d3ab4215d4de188697b4d628b985f"
format_stamp: "Formatted at 2026-08-08 15:53:42 on dist-test-slave-twwt"
server_key: "554c41adcc2072d4228a60a75c42d842"
server_key_iv: "be3a87de24391e490f748454a32a323f"
server_key_version: "encryptionkey@0"
I20260808 15:53:42.893590 25656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "8f1d3ab4215d4de188697b4d628b985f"
format_stamp: "Formatted at 2026-08-08 15:53:42 on dist-test-slave-twwt"
server_key: "554c41adcc2072d4228a60a75c42d842"
server_key_iv: "be3a87de24391e490f748454a32a323f"
server_key_version: "encryptionkey@0"
I20260808 15:53:42.899119 25656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:53:42.903115 25680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:42.904608 25656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:53:42.904822 25656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "8f1d3ab4215d4de188697b4d628b985f"
format_stamp: "Formatted at 2026-08-08 15:53:42 on dist-test-slave-twwt"
server_key: "554c41adcc2072d4228a60a75c42d842"
server_key_iv: "be3a87de24391e490f748454a32a323f"
server_key_version: "encryptionkey@0"
I20260808 15:53:42.905064 25656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:42.917114 25656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:42.921602 25656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:42.921975 25656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:42.923053 25656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:42.924459 25656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:42.924571 25656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:42.924731 25656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:42.924826 25656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:42.959838 25656 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:41557
I20260808 15:53:42.959884 25862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:41557 every 8 connection(s)
I20260808 15:53:42.961197 25656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:42.962935 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 25656
I20260808 15:53:42.963122 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:53:42.963665 27987 external_mini_cluster.cc:1428] Setting key 7f666b87e60a58fe08a04a8d7668f268
Aug 08 15:53:42 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.193@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:42.985599 25865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:42.964174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:59837 (local address 127.27.84.254:41401)
0808 15:53:42.964764 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:42.964771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:42.965848 (+ 1077us) server_negotiation.cc:407] Connection header received
0808 15:53:42.968752 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:42.968756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:42.968820 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:42.968918 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:42.971062 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:42.971802 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:42.974319 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:42.974806 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:42.974915 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:42.979422 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:42.979428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:42.979431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:42.979467 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:42.982158 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:42.983087 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:42.983092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:42.983094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:42.983235 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:42.983709 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:42.983714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:42.983719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:42.984018 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:42.984187 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:42.984960 (+ 773us) server_negotiation.cc:299] Negotiation successful
0808 15:53:42.985225 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":284,"threads_started":1}
I20260808 15:53:42.986729 25866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:42.964764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41401 (local address 127.27.84.193:59837)
0808 15:53:42.965687 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:42.965737 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:42.968435 (+ 2698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:42.969198 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:42.969212 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:42.970025 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:42.970868 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:42.970896 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:42.971970 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:42.971978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:42.974146 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:42.974169 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:42.975055 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:42.975066 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:42.975415 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:42.976228 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:42.976264 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:42.979152 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:42.982348 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:42.982357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:42.982376 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:42.982939 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:42.983361 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:42.983366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:42.983370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:42.983574 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:42.984160 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:42.984169 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:42.984542 (+ 373us) client_negotiation.cc:769] Sending connection context
0808 15:53:42.984840 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 15:53:42.985408 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":323,"threads_started":1}
I20260808 15:53:42.988546 25863 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41401
I20260808 15:53:42.988914 25863 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:42.989682 25863 heartbeater.cc:511] Master 127.27.84.254:41401 requested a full tablet report, sending...
I20260808 15:53:42.992249 25552 ts_manager.cc:194] Registered new tserver with Master: 8f1d3ab4215d4de188697b4d628b985f (127.27.84.193:41557)
I20260808 15:53:42.994565 25552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.193@KRBTEST.COM'} at 127.27.84.193:59837
WARNING: no policy specified for kudu/127.27.84.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
Entry for principal kudu/127.27.84.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab.
I20260808 15:53:43.044488 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41401
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:43.406035 25872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:43.406314 25872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:43.406409 25872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:43.413939 25872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:43.414054 25872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:43.414172 25872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:43.424036 25872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.194.keytab
--principal=kudu/127.27.84.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:43.425295 25872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:43.426622 25872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:43.440726 25881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:43.440929 25883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:43.441089 25886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:43.441862 25872 server_base.cc:1034] running on GCE node
I20260808 15:53:43.442576 25872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:43.443584 25872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:43.444813 25872 hybrid_clock.cc:648] HybridClock initialized: now 1786204423444766 us; error 31 us; skew 500 ppm
I20260808 15:53:43.445106 25872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:43 dist-test-slave-twwt krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:43.448411 25872 init.cc:377] Logged in from keytab as kudu/127.27.84.194@KRBTEST.COM (short username kudu)
I20260808 15:53:43.449996 25872 webserver.cc:460] Webserver started at http://127.27.84.194:46857/ using document root <none> and password file <none>
I20260808 15:53:43.450716 25872 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:43.450832 25872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:43.451107 25872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:43.454062 25872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "ea37251ce6524ebcbcafaf67b746a874"
format_stamp: "Formatted at 2026-08-08 15:53:43 on dist-test-slave-twwt"
server_key: "c32a8fe4387828368f807de596e86ca0"
server_key_iv: "97066de858eb2658b0acc9597e3668a1"
server_key_version: "encryptionkey@0"
I20260808 15:53:43.454871 25872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "ea37251ce6524ebcbcafaf67b746a874"
format_stamp: "Formatted at 2026-08-08 15:53:43 on dist-test-slave-twwt"
server_key: "c32a8fe4387828368f807de596e86ca0"
server_key_iv: "97066de858eb2658b0acc9597e3668a1"
server_key_version: "encryptionkey@0"
I20260808 15:53:43.460132 25872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 15:53:43.463665 25900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:43.464962 25872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:53:43.465090 25872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "ea37251ce6524ebcbcafaf67b746a874"
format_stamp: "Formatted at 2026-08-08 15:53:43 on dist-test-slave-twwt"
server_key: "c32a8fe4387828368f807de596e86ca0"
server_key_iv: "97066de858eb2658b0acc9597e3668a1"
server_key_version: "encryptionkey@0"
I20260808 15:53:43.465256 25872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:43.481164 25872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:43.485394 25872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:43.485654 25872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:43.486886 25872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:43.488281 25872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:43.488356 25872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:43.488477 25872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:43.488523 25872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:43.522219 25872 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:42703
I20260808 15:53:43.522256 26075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:42703 every 8 connection(s)
I20260808 15:53:43.523602 25872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:43.530818 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 25872
I20260808 15:53:43.531013 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:53:43.531471 27987 external_mini_cluster.cc:1428] Setting key e900a5ce1252021ca5aa57cfbcc2468a
Aug 08 15:53:43 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.194@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:43.544690 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:43.526369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:36635 (local address 127.27.84.254:41401)
0808 15:53:43.526918 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:43.526923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:43.527949 (+ 1026us) server_negotiation.cc:407] Connection header received
0808 15:53:43.530017 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:43.530023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:43.530112 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:43.530248 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:43.532074 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:43.532883 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:43.534459 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:43.534961 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:43.535097 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:43.538826 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:43.538831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:43.538835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:43.538867 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:43.541496 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:43.542231 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:43.542239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:43.542247 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:43.542443 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:43.542926 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:43.542931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:43.542935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:43.543218 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:43.543388 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:43.544085 (+ 697us) server_negotiation.cc:299] Negotiation successful
0808 15:53:43.544311 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":296,"threads_started":1}
I20260808 15:53:43.545706 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:43.527021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41401 (local address 127.27.84.194:36635)
0808 15:53:43.527725 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:43.527772 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:43.529728 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:43.530502 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:43.530510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:43.531197 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:43.531893 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:43.531913 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:43.533059 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:43.533063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:43.534309 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:43.534321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:43.535212 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:43.535219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:43.535447 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:43.536159 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:43.536186 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:43.538548 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:43.541644 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:43.541653 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:43.541669 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:43.542096 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:43.542573 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:43.542578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:43.542582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:43.542798 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:43.543359 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:43.543367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:43.543721 (+ 354us) client_negotiation.cc:769] Sending connection context
0808 15:53:43.543993 (+ 272us) client_negotiation.cc:240] Negotiation successful
0808 15:53:43.544541 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":280,"threads_started":1}
I20260808 15:53:43.547235 26077 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41401
I20260808 15:53:43.547564 26077 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:43.548394 26077 heartbeater.cc:511] Master 127.27.84.254:41401 requested a full tablet report, sending...
I20260808 15:53:43.550108 25552 ts_manager.cc:194] Registered new tserver with Master: ea37251ce6524ebcbcafaf67b746a874 (127.27.84.194:42703)
I20260808 15:53:43.551030 25552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.194@KRBTEST.COM'} at 127.27.84.194:36635
WARNING: no policy specified for kudu/127.27.84.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
Entry for principal kudu/127.27.84.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab.
I20260808 15:53:43.610369 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41401
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:34159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:43.973227 26086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:43.973726 26086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:43.973896 26086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:43.981523 26086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:43.981648 26086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:43.981767 26086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:43.991900 26086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:34159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.195.keytab
--principal=kudu/127.27.84.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:41401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:43.993266 26086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:43.994832 26086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260808 15:53:43.997843 25863 heartbeater.cc:503] Master 127.27.84.254:41401 was elected leader, sending a full tablet report...
W20260808 15:53:44.009285 26094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:44.009441 26092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:44.009290 26099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:44.010205 26086 server_base.cc:1034] running on GCE node
I20260808 15:53:44.010970 26086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:44.011986 26086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:44.013235 26086 hybrid_clock.cc:648] HybridClock initialized: now 1786204424013185 us; error 44 us; skew 500 ppm
I20260808 15:53:44.013590 26086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:44 dist-test-slave-twwt krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:44.017127 26086 init.cc:377] Logged in from keytab as kudu/127.27.84.195@KRBTEST.COM (short username kudu)
I20260808 15:53:44.018783 26086 webserver.cc:460] Webserver started at http://127.27.84.195:45429/ using document root <none> and password file <none>
I20260808 15:53:44.019526 26086 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:44.019690 26086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:44.020044 26086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:44.023319 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "fcda68e39f454d6b8e96c4d5ff624215"
format_stamp: "Formatted at 2026-08-08 15:53:44 on dist-test-slave-twwt"
server_key: "3d0d30ca509963ee8fcc522da9aab50a"
server_key_iv: "5980722eeae9aaf5258a071cb11dcb97"
server_key_version: "encryptionkey@0"
I20260808 15:53:44.024160 26086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "fcda68e39f454d6b8e96c4d5ff624215"
format_stamp: "Formatted at 2026-08-08 15:53:44 on dist-test-slave-twwt"
server_key: "3d0d30ca509963ee8fcc522da9aab50a"
server_key_iv: "5980722eeae9aaf5258a071cb11dcb97"
server_key_version: "encryptionkey@0"
I20260808 15:53:44.029870 26086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:53:44.033600 26109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:44.035208 26086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:53:44.035404 26086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "fcda68e39f454d6b8e96c4d5ff624215"
format_stamp: "Formatted at 2026-08-08 15:53:44 on dist-test-slave-twwt"
server_key: "3d0d30ca509963ee8fcc522da9aab50a"
server_key_iv: "5980722eeae9aaf5258a071cb11dcb97"
server_key_version: "encryptionkey@0"
I20260808 15:53:44.035614 26086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:44.050144 26086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:44.054080 26086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:44.054388 26086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:44.055437 26086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:44.056766 26086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:44.056882 26086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:44.057026 26086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:44.057106 26086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:44.092171 26086 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:42403
I20260808 15:53:44.092235 26284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:42403 every 8 connection(s)
I20260808 15:53:44.093587 26086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:44.095825 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 26086
I20260808 15:53:44.096036 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:53:44.096611 27987 external_mini_cluster.cc:1428] Setting key 17271ae07ab349c4a5e6780783809f20
Aug 08 15:53:44 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.195@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:53:44.116003 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.096658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:36647 (local address 127.27.84.254:41401)
0808 15:53:44.097188 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.097193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.098541 (+ 1348us) server_negotiation.cc:407] Connection header received
0808 15:53:44.100630 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.100634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.100701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.100794 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:44.102329 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.103212 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.104745 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.105155 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.105247 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.109027 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:44.109033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:44.109037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:44.109091 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:44.112717 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.113482 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.113489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.113492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.113651 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.114132 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.114138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.114141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.114421 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:44.114592 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.115338 (+ 746us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.115564 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":235,"threads_started":1}
I20260808 15:53:44.117028 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.097345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:41401 (local address 127.27.84.195:36647)
0808 15:53:44.098327 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:44.098393 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:44.100373 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:44.101017 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:44.101025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:44.101586 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:44.102170 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.102186 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.103352 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.103356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.104610 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.104622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.105374 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.105379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.105579 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.106315 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:53:44.106346 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:44.108758 (+ 2412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:44.112874 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:44.112884 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:44.112902 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:44.113355 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:44.113774 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:53:44.113778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:53:44.113782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:53:44.114011 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:53:44.114559 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:44.114567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:44.114962 (+ 395us) client_negotiation.cc:769] Sending connection context
0808 15:53:44.115247 (+ 285us) client_negotiation.cc:240] Negotiation successful
0808 15:53:44.115860 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":369,"threads_started":1}
I20260808 15:53:44.118462 26285 heartbeater.cc:347] Connected to a master server at 127.27.84.254:41401
I20260808 15:53:44.118845 26285 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:44.119618 26285 heartbeater.cc:511] Master 127.27.84.254:41401 requested a full tablet report, sending...
I20260808 15:53:44.121090 25552 ts_manager.cc:194] Registered new tserver with Master: fcda68e39f454d6b8e96c4d5ff624215 (127.27.84.195:42403)
I20260808 15:53:44.121941 25552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.84.195@KRBTEST.COM'} at 127.27.84.195:36647
I20260808 15:53:44.125417 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:53:44.144439 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.130133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.27.84.254:41401)
0808 15:53:44.130335 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.130339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.130899 (+ 560us) server_negotiation.cc:407] Connection header received
0808 15:53:44.130992 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.130996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.131073 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.131165 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:44.132721 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.133509 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.135197 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.135610 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.135712 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.138747 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:44.138753 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:44.138764 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:44.138810 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:44.141549 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.142346 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.142350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.142353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.142480 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.142991 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.142995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.142997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.143327 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:44.143517 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.144015 (+ 498us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.144207 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260808 15:53:44.148842 25552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:53:44.155905 25552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:53:44.186266 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.171928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.27.84.195:42403)
0808 15:53:44.177008 (+ 5080us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.177019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.177066 (+ 47us) server_negotiation.cc:407] Connection header received
0808 15:53:44.177193 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.177204 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.177527 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.178359 (+ 832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:44.178710 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.180230 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.183288 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.185001 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.185165 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.185460 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.185751 (+ 291us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.185928 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4876,"thread_start_us":4715,"threads_started":1}
I20260808 15:53:44.186585 26304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.169595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:42403 (local address 127.0.0.1:44658)
0808 15:53:44.173802 (+ 4207us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:44.173870 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:44.174099 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:44.177885 (+ 3786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:44.177897 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:44.177939 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:44.178494 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.178509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.180459 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.180466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.183105 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.183119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.185249 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.185256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.185468 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.185530 (+ 62us) client_negotiation.cc:769] Sending connection context
0808 15:53:44.185693 (+ 163us) client_negotiation.cc:240] Negotiation successful
0808 15:53:44.185909 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2376,"mutex_wait_us":107,"thread_start_us":5664,"threads_started":1}
I20260808 15:53:44.186697 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.169645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:42703 (local address 127.0.0.1:35180)
0808 15:53:44.175565 (+ 5920us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:44.175584 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:44.175701 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:44.178080 (+ 2379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:44.178085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:44.178113 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:44.178958 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.178972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.180811 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.180816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.183318 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.183333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.185676 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.185684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.185904 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.185940 (+ 36us) client_negotiation.cc:769] Sending connection context
0808 15:53:44.186022 (+ 82us) client_negotiation.cc:240] Negotiation successful
0808 15:53:44.186172 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5776,"mutex_wait_us":40,"thread_start_us":2597,"threads_started":1}
I20260808 15:53:44.187410 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.172050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.27.84.194:42703)
0808 15:53:44.177441 (+ 5391us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.177451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.177480 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:53:44.177605 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.177615 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.177866 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.178098 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:44.179176 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.180646 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.183498 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.185129 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.185269 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.185441 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.186885 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.187043 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5225,"thread_start_us":3887,"threads_started":1}
I20260808 15:53:44.188207 26310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.177396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41557 (local address 127.0.0.1:43118)
0808 15:53:44.178074 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:44.178094 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:44.178313 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:44.179002 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:44.179007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:44.179038 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:44.179445 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.179459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.181353 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.181362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.183890 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.183901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.187471 (+ 3570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.187476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.187634 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.187664 (+ 30us) client_negotiation.cc:769] Sending connection context
0808 15:53:44.187767 (+ 103us) client_negotiation.cc:240] Negotiation successful
0808 15:53:44.187896 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":2,"thread_start_us":326,"threads_started":1}
I20260808 15:53:44.188423 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.176084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43118 (local address 127.27.84.193:41557)
0808 15:53:44.177416 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.177424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.178196 (+ 772us) server_negotiation.cc:407] Connection header received
0808 15:53:44.178517 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.178525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.178788 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.178979 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:44.179634 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.181175 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.184059 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.185016 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.185202 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.185448 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.187856 (+ 2408us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.188047 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":969,"threads_started":1}
I20260808 15:53:44.191285 25972 tablet_service.cc:1467] Processing CreateTablet for tablet 85abec61413d425ca9bb2f839e65b37f (DEFAULT_TABLE table=test-table [id=bdb9a1cd3720497e9ba42eeabb45acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:44.191414 26180 tablet_service.cc:1467] Processing CreateTablet for tablet 85abec61413d425ca9bb2f839e65b37f (DEFAULT_TABLE table=test-table [id=bdb9a1cd3720497e9ba42eeabb45acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:44.191968 25762 tablet_service.cc:1467] Processing CreateTablet for tablet 85abec61413d425ca9bb2f839e65b37f (DEFAULT_TABLE table=test-table [id=bdb9a1cd3720497e9ba42eeabb45acef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:44.193104 26180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85abec61413d425ca9bb2f839e65b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:44.193365 25762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85abec61413d425ca9bb2f839e65b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:44.193449 25972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85abec61413d425ca9bb2f839e65b37f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:44.207620 26311 tablet_bootstrap.cc:492] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Bootstrap starting.
I20260808 15:53:44.210455 26314 tablet_bootstrap.cc:492] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Bootstrap starting.
I20260808 15:53:44.210904 26312 tablet_bootstrap.cc:492] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Bootstrap starting.
I20260808 15:53:44.213938 26312 tablet_bootstrap.cc:654] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:44.214335 26311 tablet_bootstrap.cc:654] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:44.214341 26314 tablet_bootstrap.cc:654] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:44.215327 26312 log.cc:824] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:44.215873 26311 log.cc:824] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:44.215873 26314 log.cc:824] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:44.219082 26312 tablet_bootstrap.cc:492] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: No bootstrap required, opened a new log
I20260808 15:53:44.219236 26314 tablet_bootstrap.cc:492] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: No bootstrap required, opened a new log
I20260808 15:53:44.219412 26312 ts_tablet_manager.cc:1397] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260808 15:53:44.219548 26311 tablet_bootstrap.cc:492] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: No bootstrap required, opened a new log
I20260808 15:53:44.219614 26314 ts_tablet_manager.cc:1397] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260808 15:53:44.219802 26311 ts_tablet_manager.cc:1397] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260808 15:53:44.228544 26314 raft_consensus.cc:348] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.229110 26314 raft_consensus.cc:374] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:44.229296 26314 raft_consensus.cc:729] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcda68e39f454d6b8e96c4d5ff624215, State: Initialized, Role: FOLLOWER
I20260808 15:53:44.230161 26314 consensus_queue.cc:260] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.230762 26312 raft_consensus.cc:348] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.230753 26311 raft_consensus.cc:348] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.231309 26312 raft_consensus.cc:374] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:44.231427 26312 raft_consensus.cc:729] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1d3ab4215d4de188697b4d628b985f, State: Initialized, Role: FOLLOWER
I20260808 15:53:44.231560 26285 heartbeater.cc:503] Master 127.27.84.254:41401 was elected leader, sending a full tablet report...
I20260808 15:53:44.232071 26312 consensus_queue.cc:260] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.232370 26314 ts_tablet_manager.cc:1428] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260808 15:53:44.231241 26311 raft_consensus.cc:374] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:44.232924 26311 raft_consensus.cc:729] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea37251ce6524ebcbcafaf67b746a874, State: Initialized, Role: FOLLOWER
I20260808 15:53:44.233616 26311 consensus_queue.cc:260] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.235004 26077 heartbeater.cc:503] Master 127.27.84.254:41401 was elected leader, sending a full tablet report...
I20260808 15:53:44.236554 26311 ts_tablet_manager.cc:1428] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260808 15:53:44.238313 26312 ts_tablet_manager.cc:1428] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Time spent starting tablet: real 0.019s user 0.014s sys 0.000s
W20260808 15:53:44.275915 26079 tablet.cc:2369] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:53:44.345388 26286 tablet.cc:2369] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:44.358439 26330 raft_consensus.cc:482] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:44.358848 26330 raft_consensus.cc:504] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.360821 26330 leader_election.cc:288] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcda68e39f454d6b8e96c4d5ff624215 (127.27.84.195:42403), 8f1d3ab4215d4de188697b4d628b985f (127.27.84.193:41557)
I20260808 15:53:44.368981 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.361397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:39649 (local address 127.27.84.195:42403)
0808 15:53:44.361676 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.361685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.362130 (+ 445us) server_negotiation.cc:407] Connection header received
0808 15:53:44.362384 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.362392 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.362497 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.362646 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:44.363408 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.364648 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.366884 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.367909 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.368052 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.368108 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.368544 (+ 436us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.368693 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260808 15:53:44.369127 26331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.361227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:42403 (local address 127.27.84.194:39649)
0808 15:53:44.361990 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:44.362008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:44.362212 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:44.362733 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:44.362740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:44.362772 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:44.363168 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.363207 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.364867 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.364873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.366698 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.366718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.368150 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.368157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.368354 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.368392 (+ 38us) client_negotiation.cc:769] Sending connection context
0808 15:53:44.368506 (+ 114us) client_negotiation.cc:240] Negotiation successful
0808 15:53:44.368616 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":86,"thread_start_us":375,"threads_started":1}
I20260808 15:53:44.369541 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.362085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:41557 (local address 127.27.84.194:55139)
0808 15:53:44.362995 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:44.363015 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:44.363146 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:44.363549 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:44.363553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:44.363581 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:44.364032 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.364046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.365276 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.365281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.367120 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:44.367136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.368501 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.368509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:44.368716 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.368805 (+ 89us) client_negotiation.cc:769] Sending connection context
0808 15:53:44.368904 (+ 99us) client_negotiation.cc:240] Negotiation successful
0808 15:53:44.369043 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":98,"thread_start_us":396,"threads_started":1}
I20260808 15:53:44.369560 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.362428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:55139 (local address 127.27.84.193:41557)
0808 15:53:44.362689 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.362695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.363127 (+ 432us) server_negotiation.cc:407] Connection header received
0808 15:53:44.363286 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.363294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.363378 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.363494 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:44.364216 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.365126 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.367296 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.368187 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.368380 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.368452 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.368960 (+ 508us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.369124 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 15:53:44.370390 26206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85abec61413d425ca9bb2f839e65b37f" candidate_uuid: "ea37251ce6524ebcbcafaf67b746a874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcda68e39f454d6b8e96c4d5ff624215" is_pre_election: true
I20260808 15:53:44.370739 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85abec61413d425ca9bb2f839e65b37f" candidate_uuid: "ea37251ce6524ebcbcafaf67b746a874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1d3ab4215d4de188697b4d628b985f" is_pre_election: true
I20260808 15:53:44.370973 26206 raft_consensus.cc:2454] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea37251ce6524ebcbcafaf67b746a874 in term 0.
I20260808 15:53:44.371320 25797 raft_consensus.cc:2454] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea37251ce6524ebcbcafaf67b746a874 in term 0.
I20260808 15:53:44.371882 25901 leader_election.cc:302] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea37251ce6524ebcbcafaf67b746a874, fcda68e39f454d6b8e96c4d5ff624215; no voters:
I20260808 15:53:44.372310 26330 raft_consensus.cc:2781] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:53:44.372474 26330 raft_consensus.cc:482] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:53:44.372601 26330 raft_consensus.cc:3037] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:44.374965 26330 raft_consensus.cc:504] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.375569 26330 leader_election.cc:288] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [CANDIDATE]: Term 1 election: Requested vote from peers fcda68e39f454d6b8e96c4d5ff624215 (127.27.84.195:42403), 8f1d3ab4215d4de188697b4d628b985f (127.27.84.193:41557)
I20260808 15:53:44.376119 26206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85abec61413d425ca9bb2f839e65b37f" candidate_uuid: "ea37251ce6524ebcbcafaf67b746a874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcda68e39f454d6b8e96c4d5ff624215"
I20260808 15:53:44.376260 25797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85abec61413d425ca9bb2f839e65b37f" candidate_uuid: "ea37251ce6524ebcbcafaf67b746a874" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1d3ab4215d4de188697b4d628b985f"
I20260808 15:53:44.376348 26206 raft_consensus.cc:3037] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:44.376480 25797 raft_consensus.cc:3037] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:44.379719 26206 raft_consensus.cc:2454] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea37251ce6524ebcbcafaf67b746a874 in term 1.
I20260808 15:53:44.379719 25797 raft_consensus.cc:2454] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea37251ce6524ebcbcafaf67b746a874 in term 1.
I20260808 15:53:44.380143 25901 leader_election.cc:302] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea37251ce6524ebcbcafaf67b746a874, fcda68e39f454d6b8e96c4d5ff624215; no voters:
I20260808 15:53:44.380472 26330 raft_consensus.cc:2781] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:44.380923 26330 raft_consensus.cc:686] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 1 LEADER]: Becoming Leader. State: Replica: ea37251ce6524ebcbcafaf67b746a874, State: Running, Role: LEADER
I20260808 15:53:44.381536 26330 consensus_queue.cc:237] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } }
I20260808 15:53:44.386313 25546 catalog_manager.cc:5495] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea37251ce6524ebcbcafaf67b746a874 (127.27.84.194). New cstate: current_term: 1 leader_uuid: "ea37251ce6524ebcbcafaf67b746a874" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea37251ce6524ebcbcafaf67b746a874" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 42703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 } health_report { overall_health: UNKNOWN } } }
I20260808 15:53:44.424274 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.418726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.27.84.194:42703)
0808 15:53:44.418962 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.418968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.418988 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:53:44.419189 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.419194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.419283 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.419449 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:53:44.419861 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.420602 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.422006 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.422440 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.422583 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.423137 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:53:44.423658 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:53:44.423839 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.423893 (+ 54us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.423979 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:53:44.434018 25797 raft_consensus.cc:1261] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 1 FOLLOWER]: Refusing update from remote peer ea37251ce6524ebcbcafaf67b746a874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:44.434015 26206 raft_consensus.cc:1261] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 1 FOLLOWER]: Refusing update from remote peer ea37251ce6524ebcbcafaf67b746a874: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:44.435139 26333 consensus_queue.cc:1035] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1d3ab4215d4de188697b4d628b985f" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:44.435724 26330 consensus_queue.cc:1035] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcda68e39f454d6b8e96c4d5ff624215" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 42403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:44.448920 26337 mvcc.cc:205] Tried to move back new op lower bound from 7316293322468511744 to 7316293322270830592. Current Snapshot: MvccSnapshot[applied={T|T < 7316293322468511744}]
I20260808 15:53:44.462044 25546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:44.462344 25546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:44.466300 25546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fde28bd0e8c946aa98b3073464e15d43: Sending DeleteTablet for 3 replicas of tablet 85abec61413d425ca9bb2f839e65b37f
I20260808 15:53:44.467918 25972 tablet_service.cc:1514] Processing DeleteTablet for tablet 85abec61413d425ca9bb2f839e65b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:44 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:35180
I20260808 15:53:44.467964 25762 tablet_service.cc:1514] Processing DeleteTablet for tablet 85abec61413d425ca9bb2f839e65b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:44 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:43118
I20260808 15:53:44.468775 26180 tablet_service.cc:1514] Processing DeleteTablet for tablet 85abec61413d425ca9bb2f839e65b37f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:44 UTC) from {username='kudu', principal='kudu/127.27.84.254@KRBTEST.COM'} at 127.0.0.1:44658
I20260808 15:53:44.468896 26357 tablet_replica.cc:330] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: stopping tablet replica
I20260808 15:53:44.469400 26357 raft_consensus.cc:2229] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:53:44.469762 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 25656
I20260808 15:53:44.470191 26357 raft_consensus.cc:2258] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:44.472612 26357 ts_tablet_manager.cc:1905] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:44.482460 26357 ts_tablet_manager.cc:1918] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:44.482756 26357 log.cc:1196] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/85abec61413d425ca9bb2f839e65b37f
I20260808 15:53:44.483464 26357 ts_tablet_manager.cc:1939] T 85abec61413d425ca9bb2f839e65b37f P ea37251ce6524ebcbcafaf67b746a874: Deleting consensus metadata
I20260808 15:53:44.485742 25525 catalog_manager.cc:4841] TS ea37251ce6524ebcbcafaf67b746a874 (127.27.84.194:42703): tablet 85abec61413d425ca9bb2f839e65b37f (table test-table [id=bdb9a1cd3720497e9ba42eeabb45acef]) successfully deleted
I20260808 15:53:44.486267 26359 tablet_replica.cc:330] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: stopping tablet replica
I20260808 15:53:44.486339 26360 tablet_replica.cc:330] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: stopping tablet replica
I20260808 15:53:44.486792 26359 raft_consensus.cc:2229] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:44.486963 26360 raft_consensus.cc:2229] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:44.487327 26359 raft_consensus.cc:2258] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:44.487484 26360 raft_consensus.cc:2258] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:44.489557 26359 ts_tablet_manager.cc:1905] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:44.490655 26360 ts_tablet_manager.cc:1905] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:44.495607 26359 ts_tablet_manager.cc:1918] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:44.495808 26359 log.cc:1196] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-0/wal/wals/85abec61413d425ca9bb2f839e65b37f
I20260808 15:53:44.496276 26359 ts_tablet_manager.cc:1939] T 85abec61413d425ca9bb2f839e65b37f P 8f1d3ab4215d4de188697b4d628b985f: Deleting consensus metadata
I20260808 15:53:44.498387 26360 ts_tablet_manager.cc:1918] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:44.498678 26360 log.cc:1196] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/85abec61413d425ca9bb2f839e65b37f
I20260808 15:53:44.498584 25524 catalog_manager.cc:4841] TS 8f1d3ab4215d4de188697b4d628b985f (127.27.84.193:41557): tablet 85abec61413d425ca9bb2f839e65b37f (table test-table [id=bdb9a1cd3720497e9ba42eeabb45acef]) successfully deleted
I20260808 15:53:44.499380 26360 ts_tablet_manager.cc:1939] T 85abec61413d425ca9bb2f839e65b37f P fcda68e39f454d6b8e96c4d5ff624215: Deleting consensus metadata
Aug 08 15:53:44 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.193@KRBTEST.COM
I20260808 15:53:44.501247 25524 catalog_manager.cc:4841] TS fcda68e39f454d6b8e96c4d5ff624215 (127.27.84.195:42403): tablet 85abec61413d425ca9bb2f839e65b37f (table test-table [id=bdb9a1cd3720497e9ba42eeabb45acef]) successfully deleted
I20260808 15:53:44.507349 26309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.471420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.27.84.193:41557)
0808 15:53:44.471597 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.471605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.481919 (+ 10314us) server_negotiation.cc:407] Connection header received
0808 15:53:44.482581 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.482587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.482708 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.482810 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:44.487743 (+ 4933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.489141 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.491124 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.491799 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.491902 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.501166 (+ 9264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:44.501180 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:44.501197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:44.501244 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:44.504359 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.505269 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.505278 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.505280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.505416 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.505925 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.505929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.505931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.506284 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:44.506516 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.506916 (+ 400us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.507077 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:53:44.508160 25857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:44.635865 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 25656
I20260808 15:53:44.662052 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 25872
Aug 08 15:53:44 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.194@KRBTEST.COM
I20260808 15:53:44.680320 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.662934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.27.84.194:42703)
0808 15:53:44.663125 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.663129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.663255 (+ 126us) server_negotiation.cc:407] Connection header received
0808 15:53:44.663420 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.663424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.663488 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.663579 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:44.665352 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.666219 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.668273 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.668708 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.668869 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.674087 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:44.674102 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:44.674118 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:44.674156 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:44.677240 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.678039 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.678047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.678050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.678207 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.678626 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.678634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.678637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.679036 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:44.679419 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.679706 (+ 287us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.679939 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260808 15:53:44.681087 26065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:44.802151 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 25872
I20260808 15:53:44.827921 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 26086
Aug 08 15:53:44 dist-test-slave-twwt krb5kdc[25480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.195@KRBTEST.COM
I20260808 15:53:44.843648 26306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:44.828784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44662 (local address 127.27.84.195:42403)
0808 15:53:44.828966 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:44.828970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:44.829039 (+ 69us) server_negotiation.cc:407] Connection header received
0808 15:53:44.829195 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:44.829199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:44.829265 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:44.829370 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:44.830805 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.831563 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.832902 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:44.833279 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:44.833380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:44.838026 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:44.838041 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:44.838058 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:44.838091 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:44.841113 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.841766 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.841773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.841776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.841912 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:44.842242 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:44.842245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:44.842247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:44.842596 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:44.842889 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:44.843203 (+ 314us) server_negotiation.cc:299] Negotiation successful
0808 15:53:44.843382 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:53:44.844331 26278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:44.935381 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 26086
I20260808 15:53:44.966437 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 25496
I20260808 15:53:44.967684 25600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:45.102408 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 25496
2026-08-08T15:53:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5107 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-08T15:53:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:45Z Disabled control of system clock
I20260808 15:53:45.213887 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42907
--webserver_interface=127.27.84.254
--webserver_port=0
--builtin_ntp_servers=127.27.84.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:53:45.582131 26376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:45.582458 26376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:45.582653 26376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:45.590255 26376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:45.590379 26376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:45.590445 26376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:45.590538 26376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:45.590642 26376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:45.600383 26376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42907
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:45.601768 26376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:45.603724 26376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:45.616266 26388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:45.616225 26393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:45.616243 26398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:45.617031 26376 server_base.cc:1034] running on GCE node
I20260808 15:53:45.617921 26376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:45.619520 26376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:45.620761 26376 hybrid_clock.cc:648] HybridClock initialized: now 1786204425620704 us; error 56 us; skew 500 ppm
I20260808 15:53:45.621207 26376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:45.623822 26376 webserver.cc:460] Webserver started at http://127.27.84.254:42365/ using document root <none> and password file <none>
I20260808 15:53:45.624580 26376 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:45.624724 26376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:45.625072 26376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:45.628317 26376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "407381c3e5f34fc6a6ed718f362b539a"
format_stamp: "Formatted at 2026-08-08 15:53:45 on dist-test-slave-twwt"
server_key: "d71aece6da84869f093b71c0c4dce17f"
server_key_iv: "a824f79e994ca57cc4072f0e618c1796"
server_key_version: "encryptionkey@0"
I20260808 15:53:45.629186 26376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "407381c3e5f34fc6a6ed718f362b539a"
format_stamp: "Formatted at 2026-08-08 15:53:45 on dist-test-slave-twwt"
server_key: "d71aece6da84869f093b71c0c4dce17f"
server_key_iv: "a824f79e994ca57cc4072f0e618c1796"
server_key_version: "encryptionkey@0"
I20260808 15:53:45.634886 26376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:53:45.638854 26418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:45.640424 26376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:45.640626 26376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "407381c3e5f34fc6a6ed718f362b539a"
format_stamp: "Formatted at 2026-08-08 15:53:45 on dist-test-slave-twwt"
server_key: "d71aece6da84869f093b71c0c4dce17f"
server_key_iv: "a824f79e994ca57cc4072f0e618c1796"
server_key_version: "encryptionkey@0"
I20260808 15:53:45.640858 26376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:45.684692 26376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:45.685635 26376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:45.685951 26376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:45.706367 26376 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:42907
I20260808 15:53:45.706388 26498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:42907 every 8 connection(s)
I20260808 15:53:45.708031 26376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:45.709859 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 26376
I20260808 15:53:45.710065 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:45.710568 27987 external_mini_cluster.cc:1428] Setting key fd30c6ccf0aeacb523115beaeef6cb55
I20260808 15:53:45.712782 26499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:45.726998 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:45.712901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.27.84.254:42907)
0808 15:53:45.714123 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:45.714139 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:45.714194 (+ 55us) server_negotiation.cc:407] Connection header received
0808 15:53:45.716352 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:45.716381 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:45.717029 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:45.717478 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:45.718046 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:45.720108 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:45.721453 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:45.722480 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:45.722698 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:45.723935 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:45.723950 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:45.723968 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:45.724730 (+ 762us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:45.724841 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:45.724870 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:45.725000 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:45.725276 (+ 276us) server_negotiation.cc:299] Negotiation successful
0808 15:53:45.725635 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":385,"threads_started":1}
I20260808 15:53:45.728286 26499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a: Bootstrap starting.
I20260808 15:53:45.731658 26499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:45.732656 26499 log.cc:824] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:45.735572 26499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a: No bootstrap required, opened a new log
I20260808 15:53:45.743407 26499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407381c3e5f34fc6a6ed718f362b539a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42907 } }
I20260808 15:53:45.743827 26499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:45.743933 26499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 407381c3e5f34fc6a6ed718f362b539a, State: Initialized, Role: FOLLOWER
I20260808 15:53:45.744578 26499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407381c3e5f34fc6a6ed718f362b539a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42907 } }
I20260808 15:53:45.744833 26499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:45.744997 26499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:45.745194 26499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:45.747653 26499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407381c3e5f34fc6a6ed718f362b539a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42907 } }
I20260808 15:53:45.748283 26499 leader_election.cc:302] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 407381c3e5f34fc6a6ed718f362b539a; no voters:
I20260808 15:53:45.749020 26499 leader_election.cc:288] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:45.749116 26516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:45.749922 26516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [term 1 LEADER]: Becoming Leader. State: Replica: 407381c3e5f34fc6a6ed718f362b539a, State: Running, Role: LEADER
I20260808 15:53:45.750514 26516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407381c3e5f34fc6a6ed718f362b539a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42907 } }
I20260808 15:53:45.751365 26499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:45.753479 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "407381c3e5f34fc6a6ed718f362b539a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407381c3e5f34fc6a6ed718f362b539a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42907 } } }
I20260808 15:53:45.753990 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:45.755918 26518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 407381c3e5f34fc6a6ed718f362b539a. Latest consensus state: current_term: 1 leader_uuid: "407381c3e5f34fc6a6ed718f362b539a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "407381c3e5f34fc6a6ed718f362b539a" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42907 } } }
I20260808 15:53:45.756409 26518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:45.758328 26527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:45.763628 26527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:45.772716 26527 catalog_manager.cc:1347] Generated new cluster ID: 6f992f9a380f4e0e926fd6a9e357c9f5
I20260808 15:53:45.772873 26527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:45.790627 26527 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:45.792501 26527 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:45.811281 26527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a: Generated new TSK 0
I20260808 15:53:45.812191 26527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:45.827019 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.193:0
--local_ip_for_outbound_sockets=127.27.84.193
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42907
--builtin_ntp_servers=127.27.84.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260808 15:53:46.218387 26540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:46.218756 26540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:46.218936 26540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:46.226594 26540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:46.226789 26540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:46.226980 26540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.193
I20260808 15:53:46.236737 26540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.84.193
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.193
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:46.238152 26540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:46.240192 26540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:46.254484 26546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:46.254534 26547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:46.254437 26553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:46.259074 26540 server_base.cc:1034] running on GCE node
I20260808 15:53:46.259774 26540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:46.260804 26540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:46.262058 26540 hybrid_clock.cc:648] HybridClock initialized: now 1786204426261998 us; error 49 us; skew 500 ppm
I20260808 15:53:46.262426 26540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:46.265082 26540 webserver.cc:460] Webserver started at http://127.27.84.193:37669/ using document root <none> and password file <none>
I20260808 15:53:46.265829 26540 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:46.265973 26540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:46.266317 26540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:46.269490 26540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data/instance:
uuid: "038fd10463474c5eacc5417a2e956e25"
format_stamp: "Formatted at 2026-08-08 15:53:46 on dist-test-slave-twwt"
server_key: "97e3f81c03d8a0ad8ef90306ec005797"
server_key_iv: "d07fa989a15ef8ee4151104360e4fdd5"
server_key_version: "encryptionkey@0"
I20260808 15:53:46.270344 26540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance:
uuid: "038fd10463474c5eacc5417a2e956e25"
format_stamp: "Formatted at 2026-08-08 15:53:46 on dist-test-slave-twwt"
server_key: "97e3f81c03d8a0ad8ef90306ec005797"
server_key_iv: "d07fa989a15ef8ee4151104360e4fdd5"
server_key_version: "encryptionkey@0"
I20260808 15:53:46.275920 26540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:53:46.279731 26560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:46.281116 26540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:46.281322 26540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal
uuid: "038fd10463474c5eacc5417a2e956e25"
format_stamp: "Formatted at 2026-08-08 15:53:46 on dist-test-slave-twwt"
server_key: "97e3f81c03d8a0ad8ef90306ec005797"
server_key_iv: "d07fa989a15ef8ee4151104360e4fdd5"
server_key_version: "encryptionkey@0"
I20260808 15:53:46.281564 26540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:46.296691 26540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:46.297560 26540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:46.297894 26540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:46.298975 26540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:46.300504 26540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:46.300630 26540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:46.300773 26540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:46.300871 26540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:46.335194 26540 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.193:43821
I20260808 15:53:46.335287 26738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.193:43821 every 8 connection(s)
I20260808 15:53:46.336606 26540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/data/info.pb
I20260808 15:53:46.346774 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 26540
I20260808 15:53:46.346979 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-0/wal/instance
I20260808 15:53:46.347432 27987 external_mini_cluster.cc:1428] Setting key bdc9d23629f28a87a4d3292cc62a7dbd
I20260808 15:53:46.349912 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:46.339375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:34987 (local address 127.27.84.254:42907)
0808 15:53:46.339862 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:46.339866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:46.341010 (+ 1144us) server_negotiation.cc:407] Connection header received
0808 15:53:46.342893 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:46.342899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:46.342995 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:46.343118 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:46.344243 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.345036 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.346955 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.347412 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.347568 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:46.349076 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:46.349084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:46.349091 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:46.349116 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:46.349172 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:46.349186 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:46.349313 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:46.349567 (+ 254us) server_negotiation.cc:299] Negotiation successful
0808 15:53:46.349643 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":264,"threads_started":1}
I20260808 15:53:46.350647 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.194:0
--local_ip_for_outbound_sockets=127.27.84.194
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42907
--builtin_ntp_servers=127.27.84.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:53:46.351348 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:46.339914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42907 (local address 127.27.84.193:34987)
0808 15:53:46.340850 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:46.340892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:46.342552 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:46.343364 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:46.343372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:46.343460 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:46.344036 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:46.344083 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.345210 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.345215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:46.346782 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:46.346799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.347658 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.347666 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:46.347971 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:46.348725 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:46.348727 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:46.348767 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:46.348848 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:46.349333 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:46.349344 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:46.349350 (+ 6us) client_negotiation.cc:769] Sending connection context
0808 15:53:46.349543 (+ 193us) client_negotiation.cc:240] Negotiation successful
0808 15:53:46.349953 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":272,"threads_started":1}
I20260808 15:53:46.353055 26739 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42907
I20260808 15:53:46.353374 26739 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:46.354277 26739 heartbeater.cc:511] Master 127.27.84.254:42907 requested a full tablet report, sending...
I20260808 15:53:46.356600 26443 ts_manager.cc:194] Registered new tserver with Master: 038fd10463474c5eacc5417a2e956e25 (127.27.84.193:43821)
I20260808 15:53:46.358413 26443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.193:34987
W20260808 15:53:46.721189 26747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:46.721534 26747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:46.721705 26747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:46.729398 26747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:46.729589 26747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:46.729784 26747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.194
I20260808 15:53:46.739519 26747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.84.194
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.194
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:46.740998 26747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:46.742966 26747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:46.756994 26754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:46.757040 26756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:46.757089 26760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:46.758589 26747 server_base.cc:1034] running on GCE node
I20260808 15:53:46.759440 26747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:46.760540 26747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:46.761792 26747 hybrid_clock.cc:648] HybridClock initialized: now 1786204426761727 us; error 50 us; skew 500 ppm
I20260808 15:53:46.762203 26747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:46.764869 26747 webserver.cc:460] Webserver started at http://127.27.84.194:40817/ using document root <none> and password file <none>
I20260808 15:53:46.765659 26747 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:46.765801 26747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:46.766168 26747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:46.769490 26747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data/instance:
uuid: "14bf4f10f6924a59ab4bade809a13d62"
format_stamp: "Formatted at 2026-08-08 15:53:46 on dist-test-slave-twwt"
server_key: "b18a5699ab6137b54aa5d338ad798e27"
server_key_iv: "6e0f381425b2809d48ef24e24df49f2d"
server_key_version: "encryptionkey@0"
I20260808 15:53:46.770429 26747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance:
uuid: "14bf4f10f6924a59ab4bade809a13d62"
format_stamp: "Formatted at 2026-08-08 15:53:46 on dist-test-slave-twwt"
server_key: "b18a5699ab6137b54aa5d338ad798e27"
server_key_iv: "6e0f381425b2809d48ef24e24df49f2d"
server_key_version: "encryptionkey@0"
I20260808 15:53:46.776598 26747 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260808 15:53:46.780530 26771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:46.782007 26747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:53:46.782226 26747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal
uuid: "14bf4f10f6924a59ab4bade809a13d62"
format_stamp: "Formatted at 2026-08-08 15:53:46 on dist-test-slave-twwt"
server_key: "b18a5699ab6137b54aa5d338ad798e27"
server_key_iv: "6e0f381425b2809d48ef24e24df49f2d"
server_key_version: "encryptionkey@0"
I20260808 15:53:46.782483 26747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:46.803589 26747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:46.804519 26747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:46.804858 26747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:46.805956 26747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:46.807606 26747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:46.807696 26747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:46.807859 26747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:46.807951 26747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:46.843840 26747 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.194:39663
I20260808 15:53:46.843940 26950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.194:39663 every 8 connection(s)
I20260808 15:53:46.845536 26747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/data/info.pb
I20260808 15:53:46.847040 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 26747
I20260808 15:53:46.847260 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal/instance
I20260808 15:53:46.847710 27987 external_mini_cluster.cc:1428] Setting key 9ba07cb3814b1d9f608ff9128753a40d
I20260808 15:53:46.850896 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.84.195:0
--local_ip_for_outbound_sockets=127.27.84.195
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42907
--builtin_ntp_servers=127.27.84.212:41169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260808 15:53:46.860368 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:46.848308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.194:60365 (local address 127.27.84.254:42907)
0808 15:53:46.848607 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:46.848614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:46.850548 (+ 1934us) server_negotiation.cc:407] Connection header received
0808 15:53:46.852763 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:46.852770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:46.852856 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:46.852979 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:46.854072 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.854916 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.856808 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.857539 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.857697 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:46.859399 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:46.859409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:46.859417 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:46.859444 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:46.859488 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:46.859507 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:46.859606 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:46.859840 (+ 234us) server_negotiation.cc:299] Negotiation successful
0808 15:53:46.859955 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260808 15:53:46.861582 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:46.849074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42907 (local address 127.27.84.194:60365)
0808 15:53:46.850402 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:46.850439 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:46.852469 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:46.853216 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:46.853225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:46.853317 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:46.853894 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:46.853911 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.855075 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.855080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:46.856643 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:46.856660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:46.857803 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:46.857811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:46.858213 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:46.859026 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:46.859032 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:46.859082 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:46.859181 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:46.859611 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:46.859623 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:46.859633 (+ 10us) client_negotiation.cc:769] Sending connection context
0808 15:53:46.859801 (+ 168us) client_negotiation.cc:240] Negotiation successful
0808 15:53:46.860196 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":475,"threads_started":1}
I20260808 15:53:46.863154 26951 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42907
I20260808 15:53:46.863504 26951 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:46.864456 26951 heartbeater.cc:511] Master 127.27.84.254:42907 requested a full tablet report, sending...
I20260808 15:53:46.865998 26443 ts_manager.cc:194] Registered new tserver with Master: 14bf4f10f6924a59ab4bade809a13d62 (127.27.84.194:39663)
I20260808 15:53:46.866875 26443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.194:60365
W20260808 15:53:47.211546 26955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:47.211917 26955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:47.212090 26955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:47.219601 26955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:47.219746 26955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:47.219918 26955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.84.195
I20260808 15:53:47.229400 26955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:41169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.84.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.84.195
--webserver_port=0
--tserver_master_addrs=127.27.84.254:42907
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.84.195
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:47.230984 26955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:47.232900 26955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:47.246881 26961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:47.246894 26965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:47.246826 26970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:47.248199 26955 server_base.cc:1034] running on GCE node
I20260808 15:53:47.248926 26955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:47.249992 26955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:47.251277 26955 hybrid_clock.cc:648] HybridClock initialized: now 1786204427251229 us; error 28 us; skew 500 ppm
I20260808 15:53:47.251622 26955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260808 15:53:47.254112 26955 webserver.cc:460] Webserver started at http://127.27.84.195:36769/ using document root <none> and password file <none>
I20260808 15:53:47.254998 26955 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:47.255143 26955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:47.255488 26955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:47.258966 26955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data/instance:
uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b"
format_stamp: "Formatted at 2026-08-08 15:53:47 on dist-test-slave-twwt"
server_key: "ab431cc11caa79391ce218ed9ae12e9e"
server_key_iv: "0b8dbdb7683838f9f74f153b3de0fb7c"
server_key_version: "encryptionkey@0"
I20260808 15:53:47.259879 26955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance:
uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b"
format_stamp: "Formatted at 2026-08-08 15:53:47 on dist-test-slave-twwt"
server_key: "ab431cc11caa79391ce218ed9ae12e9e"
server_key_iv: "0b8dbdb7683838f9f74f153b3de0fb7c"
server_key_version: "encryptionkey@0"
I20260808 15:53:47.266031 26955 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260808 15:53:47.270365 26981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:47.272099 26955 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260808 15:53:47.272336 26955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal
uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b"
format_stamp: "Formatted at 2026-08-08 15:53:47 on dist-test-slave-twwt"
server_key: "ab431cc11caa79391ce218ed9ae12e9e"
server_key_iv: "0b8dbdb7683838f9f74f153b3de0fb7c"
server_key_version: "encryptionkey@0"
I20260808 15:53:47.272599 26955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:47.287264 26955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:47.288164 26955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:47.288501 26955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:47.289577 26955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260808 15:53:47.291266 26955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260808 15:53:47.291393 26955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:47.291536 26955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260808 15:53:47.291626 26955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:47.325940 26955 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.195:44573
I20260808 15:53:47.325995 27160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.195:44573 every 8 connection(s)
I20260808 15:53:47.327375 26955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/data/info.pb
I20260808 15:53:47.328096 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 26955
I20260808 15:53:47.328305 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal/instance
I20260808 15:53:47.328914 27987 external_mini_cluster.cc:1428] Setting key 816936eb3680531336c832c7b0cb04b4
I20260808 15:53:47.341967 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.330331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.195:52495 (local address 127.27.84.254:42907)
0808 15:53:47.330557 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.330566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.332692 (+ 2126us) server_negotiation.cc:407] Connection header received
0808 15:53:47.334699 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.334708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.334800 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.334916 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:47.336013 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.337129 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.339009 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.339625 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.339762 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.341159 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:47.341164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:47.341168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:47.341193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:47.341228 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:47.341241 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:47.341313 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.341577 (+ 264us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.341668 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:53:47.343291 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.331143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42907 (local address 127.27.84.195:52495)
0808 15:53:47.332324 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.332363 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:47.334388 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:47.335102 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:47.335110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:47.335204 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:53:47.335839 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.335858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.337296 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.337300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.338857 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.338871 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.339886 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.339893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.340106 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.340848 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0808 15:53:47.340851 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:53:47.340894 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0808 15:53:47.340975 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:53:47.341337 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:53:47.341346 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:53:47.341353 (+ 7us) client_negotiation.cc:769] Sending connection context
0808 15:53:47.341510 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 15:53:47.341839 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":311,"threads_started":1}
I20260808 15:53:47.344820 27161 heartbeater.cc:347] Connected to a master server at 127.27.84.254:42907
I20260808 15:53:47.345155 27161 heartbeater.cc:464] Registering TS with master...
I20260808 15:53:47.346030 27161 heartbeater.cc:511] Master 127.27.84.254:42907 requested a full tablet report, sending...
I20260808 15:53:47.347535 26443 ts_manager.cc:194] Registered new tserver with Master: e98a6a27b7ce49fb92a35bd9ed0f641b (127.27.84.195:44573)
I20260808 15:53:47.348368 26443 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.84.195:52495
I20260808 15:53:47.357548 27987 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260808 15:53:47.361416 26739 heartbeater.cc:503] Master 127.27.84.254:42907 was elected leader, sending a full tablet report...
I20260808 15:53:47.368247 26745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.362072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.27.84.254:42907)
0808 15:53:47.362300 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.362306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.362928 (+ 622us) server_negotiation.cc:407] Connection header received
0808 15:53:47.363132 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.363136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.363197 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.363303 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:47.363794 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.364508 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.365817 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.366257 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.366401 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.367533 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:47.367538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:47.367541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:47.367558 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:47.367589 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:47.367601 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:47.367699 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.367846 (+ 147us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.367936 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:53:47.372208 26443 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260808 15:53:47.379503 26443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260808 15:53:47.417448 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.393267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.27.84.194:39663)
0808 15:53:47.406849 (+ 13582us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.406865 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.406911 (+ 46us) server_negotiation.cc:407] Connection header received
0808 15:53:47.407060 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.407073 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.407457 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.408517 (+ 1060us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:47.409944 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.411571 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.414712 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.416148 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.416309 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.416575 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.416874 (+ 299us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.417058 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13318,"thread_start_us":13154,"threads_started":1}
I20260808 15:53:47.417495 27177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.393295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:39663 (local address 127.0.0.1:56974)
0808 15:53:47.405940 (+ 12645us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.405972 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:47.406133 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:47.408827 (+ 2694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:47.408838 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:47.408874 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:47.409661 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.409675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.411749 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.411756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.414500 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.414511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.416338 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.416345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.416578 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.416679 (+ 101us) client_negotiation.cc:769] Sending connection context
0808 15:53:47.416836 (+ 157us) client_negotiation.cc:240] Negotiation successful
0808 15:53:47.417032 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12434,"mutex_wait_us":223,"thread_start_us":11204,"threads_started":1}
I20260808 15:53:47.417541 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.393221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:44573 (local address 127.0.0.1:52860)
0808 15:53:47.403838 (+ 10617us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.403920 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:47.404229 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:47.408882 (+ 4653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:47.408887 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:47.408914 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:47.409661 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.409675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.411374 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.411380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.414189 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.414204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.416565 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.416571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.416832 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.416907 (+ 75us) client_negotiation.cc:769] Sending connection context
0808 15:53:47.416991 (+ 84us) client_negotiation.cc:240] Negotiation successful
0808 15:53:47.417124 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10172,"mutex_wait_us":136,"thread_start_us":12334,"threads_started":1}
I20260808 15:53:47.417622 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.406645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.27.84.193:43821)
0808 15:53:47.407359 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.407368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.408150 (+ 782us) server_negotiation.cc:407] Connection header received
0808 15:53:47.408444 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.408453 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.408706 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.408908 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:47.410130 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.411657 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.414386 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.415976 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.416092 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.416399 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.417076 (+ 677us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.417251 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":390,"threads_started":1}
I20260808 15:53:47.418257 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.407416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48776)
0808 15:53:47.408020 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.408044 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:47.408260 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:47.409165 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:47.409169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:47.409194 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:47.409924 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.409937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.411823 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.411829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.414101 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.414121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.416580 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.416586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.416828 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.416900 (+ 72us) client_negotiation.cc:769] Sending connection context
0808 15:53:47.417806 (+ 906us) client_negotiation.cc:240] Negotiation successful
0808 15:53:47.417957 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":286,"thread_start_us":325,"threads_started":1}
I20260808 15:53:47.418584 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.393630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52860 (local address 127.27.84.195:44573)
0808 15:53:47.408219 (+ 14589us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.408227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.408264 (+ 37us) server_negotiation.cc:407] Connection header received
0808 15:53:47.408395 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.408403 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.408658 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.409290 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:47.409921 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.411192 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.414399 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.415976 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.416106 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.416404 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.418015 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.418184 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14409,"thread_start_us":14238,"threads_started":1}
I20260808 15:53:47.421878 26641 tablet_service.cc:1467] Processing CreateTablet for tablet a5aaaa79e2164d808847632adca00b67 (DEFAULT_TABLE table=test-table [id=8161da14f6ba4a39a63a298aeae5585c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:47.422174 27065 tablet_service.cc:1467] Processing CreateTablet for tablet a5aaaa79e2164d808847632adca00b67 (DEFAULT_TABLE table=test-table [id=8161da14f6ba4a39a63a298aeae5585c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:47.422174 26856 tablet_service.cc:1467] Processing CreateTablet for tablet a5aaaa79e2164d808847632adca00b67 (DEFAULT_TABLE table=test-table [id=8161da14f6ba4a39a63a298aeae5585c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260808 15:53:47.423513 26641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5aaaa79e2164d808847632adca00b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:47.423656 26856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5aaaa79e2164d808847632adca00b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:47.423774 27065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5aaaa79e2164d808847632adca00b67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:47.435803 27187 tablet_bootstrap.cc:492] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: Bootstrap starting.
I20260808 15:53:47.438761 27189 tablet_bootstrap.cc:492] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Bootstrap starting.
I20260808 15:53:47.439730 27187 tablet_bootstrap.cc:654] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:47.442001 27187 log.cc:824] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:47.442818 27191 tablet_bootstrap.cc:492] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Bootstrap starting.
I20260808 15:53:47.443305 27189 tablet_bootstrap.cc:654] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:47.444614 27189 log.cc:824] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:47.445530 27187 tablet_bootstrap.cc:492] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: No bootstrap required, opened a new log
I20260808 15:53:47.445865 27187 ts_tablet_manager.cc:1397] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260808 15:53:47.446542 27191 tablet_bootstrap.cc:654] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:47.447546 27191 log.cc:824] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:47.448082 27189 tablet_bootstrap.cc:492] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: No bootstrap required, opened a new log
I20260808 15:53:47.448432 27189 ts_tablet_manager.cc:1397] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260808 15:53:47.450389 27191 tablet_bootstrap.cc:492] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: No bootstrap required, opened a new log
I20260808 15:53:47.450732 27191 ts_tablet_manager.cc:1397] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260808 15:53:47.456920 27187 raft_consensus.cc:348] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.457441 27187 raft_consensus.cc:374] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:47.457556 27187 raft_consensus.cc:729] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038fd10463474c5eacc5417a2e956e25, State: Initialized, Role: FOLLOWER
I20260808 15:53:47.458407 27187 consensus_queue.cc:260] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.458562 27191 raft_consensus.cc:348] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.459103 27191 raft_consensus.cc:374] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:47.459241 27191 raft_consensus.cc:729] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e98a6a27b7ce49fb92a35bd9ed0f641b, State: Initialized, Role: FOLLOWER
I20260808 15:53:47.459404 27189 raft_consensus.cc:348] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.459981 27189 raft_consensus.cc:374] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:47.459836 27191 consensus_queue.cc:260] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.460098 27189 raft_consensus.cc:729] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14bf4f10f6924a59ab4bade809a13d62, State: Initialized, Role: FOLLOWER
I20260808 15:53:47.460809 27189 consensus_queue.cc:260] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.465672 27189 ts_tablet_manager.cc:1428] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260808 15:53:47.467720 27161 heartbeater.cc:503] Master 127.27.84.254:42907 was elected leader, sending a full tablet report...
I20260808 15:53:47.468510 27191 ts_tablet_manager.cc:1428] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260808 15:53:47.470965 26951 heartbeater.cc:503] Master 127.27.84.254:42907 was elected leader, sending a full tablet report...
I20260808 15:53:47.473435 27187 ts_tablet_manager.cc:1428] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: Time spent starting tablet: real 0.027s user 0.010s sys 0.009s
W20260808 15:53:47.579290 27163 tablet.cc:2369] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:47.586066 27196 raft_consensus.cc:482] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260808 15:53:47.586359 27196 raft_consensus.cc:504] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.588212 27196 leader_election.cc:288] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14bf4f10f6924a59ab4bade809a13d62 (127.27.84.194:39663), e98a6a27b7ce49fb92a35bd9ed0f641b (127.27.84.195:44573)
W20260808 15:53:47.589236 26742 tablet.cc:2369] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260808 15:53:47.597802 26952 tablet.cc:2369] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:53:47.598239 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.588683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:57273 (local address 127.27.84.195:44573)
0808 15:53:47.588955 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.588963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.589814 (+ 851us) server_negotiation.cc:407] Connection header received
0808 15:53:47.590027 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.590032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.590117 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.590220 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:47.591017 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.592094 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.595650 (+ 3556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.596933 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.597125 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.597186 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.597839 (+ 653us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.597986 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260808 15:53:47.598294 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.588825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.84.193:37225 (local address 127.27.84.194:39663)
0808 15:53:47.589683 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.589691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.589715 (+ 24us) server_negotiation.cc:407] Connection header received
0808 15:53:47.589799 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.589804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.589899 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.589989 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0808 15:53:47.590884 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.592069 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.595650 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.596936 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.597109 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.597186 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.597853 (+ 667us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.598010 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713}
I20260808 15:53:47.598496 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.588509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.194:39663 (local address 127.27.84.193:37225)
0808 15:53:47.589245 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.589268 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:47.589568 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:47.590148 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:47.590152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:47.590196 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:47.590688 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.590703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.592241 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.592247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.595381 (+ 3134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.595393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.597260 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.597268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.597536 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.597638 (+ 102us) client_negotiation.cc:769] Sending connection context
0808 15:53:47.597747 (+ 109us) client_negotiation.cc:240] Negotiation successful
0808 15:53:47.597909 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":416,"thread_start_us":342,"threads_started":1}
I20260808 15:53:47.599063 27204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.588506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.195:44573 (local address 127.27.84.193:57273)
0808 15:53:47.589682 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.589700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0808 15:53:47.589869 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:53:47.590244 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:53:47.590251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:53:47.590339 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0808 15:53:47.590838 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.590852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.592242 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.592247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.595373 (+ 3126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:53:47.595393 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.597228 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.597244 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:53:47.597534 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.597631 (+ 97us) client_negotiation.cc:769] Sending connection context
0808 15:53:47.598463 (+ 832us) client_negotiation.cc:240] Negotiation successful
0808 15:53:47.598672 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":692,"spinlock_wait_cycles":27520,"thread_start_us":341,"threads_started":1}
I20260808 15:53:47.599889 26886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5aaaa79e2164d808847632adca00b67" candidate_uuid: "038fd10463474c5eacc5417a2e956e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14bf4f10f6924a59ab4bade809a13d62" is_pre_election: true
I20260808 15:53:47.600193 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5aaaa79e2164d808847632adca00b67" candidate_uuid: "038fd10463474c5eacc5417a2e956e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" is_pre_election: true
I20260808 15:53:47.600590 26886 raft_consensus.cc:2454] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038fd10463474c5eacc5417a2e956e25 in term 0.
I20260808 15:53:47.600793 27093 raft_consensus.cc:2454] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038fd10463474c5eacc5417a2e956e25 in term 0.
I20260808 15:53:47.601514 26572 leader_election.cc:302] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038fd10463474c5eacc5417a2e956e25, 14bf4f10f6924a59ab4bade809a13d62; no voters:
I20260808 15:53:47.602063 27196 raft_consensus.cc:2781] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260808 15:53:47.602257 27196 raft_consensus.cc:482] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260808 15:53:47.602393 27196 raft_consensus.cc:3037] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:47.604892 27196 raft_consensus.cc:504] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.605574 27196 leader_election.cc:288] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [CANDIDATE]: Term 1 election: Requested vote from peers 14bf4f10f6924a59ab4bade809a13d62 (127.27.84.194:39663), e98a6a27b7ce49fb92a35bd9ed0f641b (127.27.84.195:44573)
I20260808 15:53:47.606063 27093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5aaaa79e2164d808847632adca00b67" candidate_uuid: "038fd10463474c5eacc5417a2e956e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b"
I20260808 15:53:47.606063 26886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5aaaa79e2164d808847632adca00b67" candidate_uuid: "038fd10463474c5eacc5417a2e956e25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14bf4f10f6924a59ab4bade809a13d62"
I20260808 15:53:47.606287 26886 raft_consensus.cc:3037] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:47.606287 27093 raft_consensus.cc:3037] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:47.609642 27093 raft_consensus.cc:2454] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038fd10463474c5eacc5417a2e956e25 in term 1.
I20260808 15:53:47.609642 26886 raft_consensus.cc:2454] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038fd10463474c5eacc5417a2e956e25 in term 1.
I20260808 15:53:47.610126 26572 leader_election.cc:302] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 038fd10463474c5eacc5417a2e956e25, 14bf4f10f6924a59ab4bade809a13d62, e98a6a27b7ce49fb92a35bd9ed0f641b; no voters:
I20260808 15:53:47.610504 27196 raft_consensus.cc:2781] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:47.611022 27196 raft_consensus.cc:686] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 1 LEADER]: Becoming Leader. State: Replica: 038fd10463474c5eacc5417a2e956e25, State: Running, Role: LEADER
I20260808 15:53:47.611532 27196 consensus_queue.cc:237] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } }
I20260808 15:53:47.616786 26443 catalog_manager.cc:5495] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038fd10463474c5eacc5417a2e956e25 (127.27.84.193). New cstate: current_term: 1 leader_uuid: "038fd10463474c5eacc5417a2e956e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038fd10463474c5eacc5417a2e956e25" member_type: VOTER last_known_addr { host: "127.27.84.193" port: 43821 } health_report { overall_health: HEALTHY } } }
I20260808 15:53:47.663205 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.657274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48792 (local address 127.27.84.193:43821)
0808 15:53:47.657476 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.657480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.657499 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:53:47.657595 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.657599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.657695 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.657831 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:53:47.658218 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.659102 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.660711 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.661339 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.661469 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.662036 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:53:47.662570 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:53:47.662804 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.662860 (+ 56us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.662943 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260808 15:53:47.671715 26886 raft_consensus.cc:1261] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 1 FOLLOWER]: Refusing update from remote peer 038fd10463474c5eacc5417a2e956e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:47.671711 27093 raft_consensus.cc:1261] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 1 FOLLOWER]: Refusing update from remote peer 038fd10463474c5eacc5417a2e956e25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:53:47.672760 27205 consensus_queue.cc:1035] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14bf4f10f6924a59ab4bade809a13d62" member_type: VOTER last_known_addr { host: "127.27.84.194" port: 39663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:47.673133 27196 consensus_queue.cc:1035] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e98a6a27b7ce49fb92a35bd9ed0f641b" member_type: VOTER last_known_addr { host: "127.27.84.195" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:53:47.700783 26443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:47.701025 26443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260808 15:53:47.705108 26443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 407381c3e5f34fc6a6ed718f362b539a: Sending DeleteTablet for 3 replicas of tablet a5aaaa79e2164d808847632adca00b67
I20260808 15:53:47.706411 26856 tablet_service.cc:1514] Processing DeleteTablet for tablet a5aaaa79e2164d808847632adca00b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:47 UTC) from {username='slave'} at 127.0.0.1:56974
I20260808 15:53:47.707247 27065 tablet_service.cc:1514] Processing DeleteTablet for tablet a5aaaa79e2164d808847632adca00b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:47 UTC) from {username='slave'} at 127.0.0.1:52860
I20260808 15:53:47.708490 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 26540
I20260808 15:53:47.710932 26641 tablet_service.cc:1514] Processing DeleteTablet for tablet a5aaaa79e2164d808847632adca00b67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-08 15:53:47 UTC) from {username='slave'} at 127.0.0.1:48776
I20260808 15:53:47.729064 27225 tablet_replica.cc:330] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25: stopping tablet replica
I20260808 15:53:47.731758 27222 tablet_replica.cc:330] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: stopping tablet replica
I20260808 15:53:47.732618 27222 raft_consensus.cc:2229] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:47.733384 27222 raft_consensus.cc:2258] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:47.736047 27222 ts_tablet_manager.cc:1905] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:47.738575 27225 raft_consensus.cc:2229] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 1 LEADER]: Raft consensus shutting down.
I20260808 15:53:47.739377 27221 tablet_replica.cc:330] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: stopping tablet replica
I20260808 15:53:47.740243 27225 raft_consensus.cc:2258] T a5aaaa79e2164d808847632adca00b67 P 038fd10463474c5eacc5417a2e956e25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:47.740878 27221 raft_consensus.cc:2229] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260808 15:53:47.740372 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.709861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.27.84.193:43821)
0808 15:53:47.710060 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.710066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.731336 (+ 21270us) server_negotiation.cc:407] Connection header received
0808 15:53:47.731847 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.731856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.731979 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.732073 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:47.732907 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.734148 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.736107 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.736771 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.736908 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.738456 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:47.738470 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:47.738489 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:47.739339 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:47.739405 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:47.739431 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:47.739514 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.739783 (+ 269us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.739912 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:53:47.741935 26731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:47.742457 27221 raft_consensus.cc:2258] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260808 15:53:47.743773 27222 ts_tablet_manager.cc:1918] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:47.744035 27222 log.cc:1196] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-2/wal/wals/a5aaaa79e2164d808847632adca00b67
I20260808 15:53:47.744725 27222 ts_tablet_manager.cc:1939] T a5aaaa79e2164d808847632adca00b67 P e98a6a27b7ce49fb92a35bd9ed0f641b: Deleting consensus metadata
I20260808 15:53:47.745651 27221 ts_tablet_manager.cc:1905] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260808 15:53:47.746950 26420 catalog_manager.cc:4841] TS e98a6a27b7ce49fb92a35bd9ed0f641b (127.27.84.195:44573): tablet a5aaaa79e2164d808847632adca00b67 (table test-table [id=8161da14f6ba4a39a63a298aeae5585c]) successfully deleted
I20260808 15:53:47.754042 27221 ts_tablet_manager.cc:1918] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260808 15:53:47.754262 27221 log.cc:1196] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Deleting WAL directory at /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786204089698720-27987-0/minicluster-data/ts-1/wal/wals/a5aaaa79e2164d808847632adca00b67
I20260808 15:53:47.754921 27221 ts_tablet_manager.cc:1939] T a5aaaa79e2164d808847632adca00b67 P 14bf4f10f6924a59ab4bade809a13d62: Deleting consensus metadata
I20260808 15:53:47.756742 26423 catalog_manager.cc:4841] TS 14bf4f10f6924a59ab4bade809a13d62 (127.27.84.194:39663): tablet a5aaaa79e2164d808847632adca00b67 (table test-table [id=8161da14f6ba4a39a63a298aeae5585c]) successfully deleted
I20260808 15:53:47.853507 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 26540
I20260808 15:53:47.887416 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 26747
W20260808 15:53:47.887336 26420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.84.193:43821 (error 108)
I20260808 15:53:47.890798 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.890302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48818)
0808 15:53:47.890478 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.890589 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260808 15:53:47.891085 26420 catalog_manager.cc:4584] TS 038fd10463474c5eacc5417a2e956e25 (127.27.84.193:43821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5aaaa79e2164d808847632adca00b67: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
I20260808 15:53:47.895062 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.888341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.27.84.194:39663)
0808 15:53:47.888686 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:47.888693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:47.888718 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:53:47.888998 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:47.889007 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:47.889144 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:47.889289 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:47.889687 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.890476 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.891946 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:47.892372 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:47.892516 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:47.893566 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:47.893575 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:47.893589 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:47.894183 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:47.894231 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:47.894250 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:47.894398 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:47.894532 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 15:53:47.894783 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260808 15:53:47.895756 26945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:47.944751 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.944130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48824)
0808 15:53:47.944449 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.944564 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260808 15:53:47.946074 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:47.945664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48830)
0808 15:53:47.945834 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:47.945936 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260808 15:53:47.990725 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 26747
I20260808 15:53:48.006852 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:48.006240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48846)
0808 15:53:48.006472 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:48.006568 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260808 15:53:48.007936 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:48.007576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48862)
0808 15:53:48.007731 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:48.007818 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260808 15:53:48.021483 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 26955
I20260808 15:53:48.028813 27183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:48.022405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52876 (local address 127.27.84.195:44573)
0808 15:53:48.022648 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:48.022656 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:48.022681 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:53:48.022943 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:48.022948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:48.023042 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:48.023159 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:48.023604 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:48.024356 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:48.025632 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:48.026034 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:48.026183 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:48.027399 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:48.027409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:48.027433 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:48.028024 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:48.028073 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:48.028094 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:48.028227 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:48.028367 (+ 140us) server_negotiation.cc:299] Negotiation successful
0808 15:53:48.028486 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260808 15:53:48.029501 27155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:48.088932 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:48.088369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48874)
0808 15:53:48.088628 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:48.088734 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260808 15:53:48.090176 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:48.089718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48882)
0808 15:53:48.089962 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:48.090040 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260808 15:53:48.120357 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 26955
I20260808 15:53:48.151113 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 26376
I20260808 15:53:48.152320 26493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:48.264254 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 26376
I20260808 15:53:48.264318 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:48.263358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.193:43821 (local address 127.0.0.1:48892)
0808 15:53:48.263579 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0808 15:53:48.263686 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.193:43821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
2026-08-08T15:53:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3158 ms)
[----------] 6 tests from EncryptionPolicyTest (24212 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:48 dist-test-slave-twwt krb5kdc[27238](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:48 dist-test-slave-twwt krb5kdc[27238](info): set up 2 sockets
Aug 08 15:53:48 dist-test-slave-twwt krb5kdc[27238](info): commencing operation
krb5kdc: starting...
W20260808 15:53:49.640975 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:49 dist-test-slave-twwt krb5kdc[27238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:53:50.012450 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:50.365084 27260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:50.365501 27260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:50.365679 27260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:50.373132 27260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:50.373302 27260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:50.373420 27260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:50.373533 27260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:50.373612 27260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:50.383710 27260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40127
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:50.385221 27260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:50.386639 27260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:50.400050 27266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:50.400107 27275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:50.400014 27272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:50.401239 27260 server_base.cc:1034] running on GCE node
I20260808 15:53:50.402204 27260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:50.403936 27260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:50.405208 27260 hybrid_clock.cc:648] HybridClock initialized: now 1786204430405093 us; error 132 us; skew 500 ppm
I20260808 15:53:50.405628 27260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:50 dist-test-slave-twwt krb5kdc[27238](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:50.412844 27260 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:53:50.414470 27260 webserver.cc:460] Webserver started at http://127.0.0.1:38277/ using document root <none> and password file <none>
I20260808 15:53:50.415246 27260 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:50.415406 27260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:50.415750 27260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:50.418906 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "2c9253212a254ed0b01c97be34c56ef6"
format_stamp: "Formatted at 2026-08-08 15:53:50 on dist-test-slave-twwt"
server_key: "87c79067479d79ce57ccd3d981c75c90"
server_key_iv: "d79a5786c83681ee38d1f59e55916b86"
server_key_version: "encryptionkey@0"
I20260808 15:53:50.419734 27260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "2c9253212a254ed0b01c97be34c56ef6"
format_stamp: "Formatted at 2026-08-08 15:53:50 on dist-test-slave-twwt"
server_key: "87c79067479d79ce57ccd3d981c75c90"
server_key_iv: "d79a5786c83681ee38d1f59e55916b86"
server_key_version: "encryptionkey@0"
I20260808 15:53:50.425412 27260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:53:50.429229 27283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:50.430720 27260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:53:50.430928 27260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "2c9253212a254ed0b01c97be34c56ef6"
format_stamp: "Formatted at 2026-08-08 15:53:50 on dist-test-slave-twwt"
server_key: "87c79067479d79ce57ccd3d981c75c90"
server_key_iv: "d79a5786c83681ee38d1f59e55916b86"
server_key_version: "encryptionkey@0"
I20260808 15:53:50.431157 27260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:50.447172 27260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:50.451141 27260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:50.451433 27260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:50.470937 27260 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40127
I20260808 15:53:50.470960 27352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40127 every 8 connection(s)
I20260808 15:53:50.472411 27260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:50.476511 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:50.477406 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 27260
I20260808 15:53:50.477635 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:50.478509 27987 external_mini_cluster.cc:1428] Setting key adedba4d6db753e47de6f9f3abed76ba
I20260808 15:53:50.491698 27354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6: Bootstrap starting.
Aug 08 15:53:50 dist-test-slave-twwt krb5kdc[27238](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:53:50.495522 27354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:50.496627 27354 log.cc:824] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:50.500221 27354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6: No bootstrap required, opened a new log
I20260808 15:53:50.503794 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:50.480694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52499 (local address 127.0.0.1:40127)
0808 15:53:50.481547 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:50.481559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:50.481600 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:53:50.483348 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:50.483374 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:50.483780 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:50.484116 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:50.485926 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:50.487277 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:50.489157 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:50.489850 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:50.489976 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:50.494985 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:50.494993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:50.495008 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:50.495049 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:50.498680 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:50.499801 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:50.499811 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:50.499820 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:50.500047 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:50.500547 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:50.500552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:50.500555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:50.501321 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:50.501656 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:50.502004 (+ 348us) server_negotiation.cc:299] Negotiation successful
0808 15:53:50.502371 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":398,"threads_started":1}
I20260808 15:53:50.507854 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:53:50.509997 27354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9253212a254ed0b01c97be34c56ef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40127 } }
I20260808 15:53:50.510470 27354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:50.510584 27354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9253212a254ed0b01c97be34c56ef6, State: Initialized, Role: FOLLOWER
I20260808 15:53:50.511292 27354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9253212a254ed0b01c97be34c56ef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40127 } }
I20260808 15:53:50.511627 27354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:50.511790 27354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:50.512004 27354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:50.516124 27354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9253212a254ed0b01c97be34c56ef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40127 } }
I20260808 15:53:50.516806 27354 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c9253212a254ed0b01c97be34c56ef6; no voters:
I20260808 15:53:50.517486 27354 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:50.517623 27374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:50.519774 27374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c9253212a254ed0b01c97be34c56ef6, State: Running, Role: LEADER
I20260808 15:53:50.519870 27354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:50.520430 27374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9253212a254ed0b01c97be34c56ef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40127 } }
I20260808 15:53:50.532311 27375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c9253212a254ed0b01c97be34c56ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9253212a254ed0b01c97be34c56ef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40127 } } }
I20260808 15:53:50.532325 27376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9253212a254ed0b01c97be34c56ef6. Latest consensus state: current_term: 1 leader_uuid: "2c9253212a254ed0b01c97be34c56ef6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9253212a254ed0b01c97be34c56ef6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40127 } } }
I20260808 15:53:50.532790 27376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:50.532788 27375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:50.536883 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:50.516528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54429 (local address 127.0.0.1:40127)
0808 15:53:50.516793 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:50.516802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:50.517214 (+ 412us) server_negotiation.cc:407] Connection header received
0808 15:53:50.517464 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:50.517469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:50.517559 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:50.517672 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:50.520244 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:50.521546 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:50.523841 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:50.524755 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:50.524924 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:50.528705 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:50.528709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:50.528712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:50.528748 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:50.532796 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:50.533958 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:50.533963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:50.533968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:50.534131 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:50.534993 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:50.534999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:50.535002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:50.535406 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:50.535602 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:50.536407 (+ 805us) server_negotiation.cc:299] Negotiation successful
0808 15:53:50.536602 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 15:53:50.538682 27398 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:53:50.538805 27398 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:53:50.538930 27399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:50.541006 27399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:50.550076 27399 catalog_manager.cc:1347] Generated new cluster ID: 1493de18f28d4a1ba671f8e34c09b263
I20260808 15:53:50.550220 27399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:50.571177 27399 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:50.573283 27399 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:50.591450 27399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c9253212a254ed0b01c97be34c56ef6: Generated new TSK 0
I20260808 15:53:50.592680 27399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:50.609242 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27260
I20260808 15:53:50.611169 27348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:50.707168 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 27260
W20260808 15:53:50.735118 27233 connection.cc:535] client connection to 127.0.0.1:40127 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40127): Connection reset by peer (error 104)
2026-08-08T15:53:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2438 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:50 dist-test-slave-twwt krb5kdc[27408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:50 dist-test-slave-twwt krb5kdc[27408](info): set up 2 sockets
Aug 08 15:53:50 dist-test-slave-twwt krb5kdc[27408](info): commencing operation
krb5kdc: starting...
W20260808 15:53:52.066366 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.001s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:52 dist-test-slave-twwt krb5kdc[27408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:53:52.421921 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40669
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40669
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:52.774964 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:52.775280 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:52.775463 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:52.782919 27425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:52.783051 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:52.783175 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:52.783309 27425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:52.783392 27425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:52.793468 27425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40669
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:52.794860 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:52.796257 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:52.808894 27432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:52.808996 27431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:52.808888 27435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:52.809798 27425 server_base.cc:1034] running on GCE node
I20260808 15:53:52.810556 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:52.812027 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:52.813258 27425 hybrid_clock.cc:648] HybridClock initialized: now 1786204432813189 us; error 51 us; skew 500 ppm
I20260808 15:53:52.813658 27425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:52 dist-test-slave-twwt krb5kdc[27408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:52.816967 27425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:53:52.818439 27425 webserver.cc:460] Webserver started at http://127.0.0.1:34389/ using document root <none> and password file <none>
I20260808 15:53:52.819176 27425 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:52.819288 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:52.819550 27425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:52.822533 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "082c2267eab74e5baf96311e91d70949"
format_stamp: "Formatted at 2026-08-08 15:53:52 on dist-test-slave-twwt"
server_key: "c5b00dadb0a5c4ab69ed12aef5fd0ac5"
server_key_iv: "a0a43def75cd2a038be39f6b4f8400ba"
server_key_version: "encryptionkey@0"
I20260808 15:53:52.823359 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "082c2267eab74e5baf96311e91d70949"
format_stamp: "Formatted at 2026-08-08 15:53:52 on dist-test-slave-twwt"
server_key: "c5b00dadb0a5c4ab69ed12aef5fd0ac5"
server_key_iv: "a0a43def75cd2a038be39f6b4f8400ba"
server_key_version: "encryptionkey@0"
I20260808 15:53:52.828569 27425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:53:52.832183 27452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:52.833607 27425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:53:52.833847 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "082c2267eab74e5baf96311e91d70949"
format_stamp: "Formatted at 2026-08-08 15:53:52 on dist-test-slave-twwt"
server_key: "c5b00dadb0a5c4ab69ed12aef5fd0ac5"
server_key_iv: "a0a43def75cd2a038be39f6b4f8400ba"
server_key_version: "encryptionkey@0"
I20260808 15:53:52.834074 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:52.852640 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:52.857139 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:52.857389 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:52.876518 27425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40669
I20260808 15:53:52.876550 27524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40669 every 8 connection(s)
I20260808 15:53:52.877928 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:52.881771 27525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:52.885659 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 27425
I20260808 15:53:52.885857 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:52.886413 27987 external_mini_cluster.cc:1428] Setting key ef9a27879a8fee8143c73884dfd720ef
I20260808 15:53:52.895272 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949: Bootstrap starting.
I20260808 15:53:52.899068 27525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:52.900290 27525 log.cc:824] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:52.901562 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:52.888412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54361 (local address 127.0.0.1:40669)
0808 15:53:52.889370 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:52.889382 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:52.889426 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:53:52.891363 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:52.891387 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:52.891867 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:52.892222 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:52.892855 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:52.894290 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:52.896176 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:52.896976 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:52.897123 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:52.898506 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:52.898514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:52.898529 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:52.899315 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:52.899408 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:52.899438 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:52.899577 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:52.899962 (+ 385us) server_negotiation.cc:299] Negotiation successful
0808 15:53:52.900177 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":477,"threads_started":1}
I20260808 15:53:52.903424 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949: No bootstrap required, opened a new log
I20260808 15:53:52.905761 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:53:52.913518 27525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082c2267eab74e5baf96311e91d70949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40669 } }
I20260808 15:53:52.914023 27525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:52.914152 27525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082c2267eab74e5baf96311e91d70949, State: Initialized, Role: FOLLOWER
I20260808 15:53:52.914837 27525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082c2267eab74e5baf96311e91d70949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40669 } }
I20260808 15:53:52.915081 27525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:52.915256 27525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:52.915428 27525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:52.918314 27525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082c2267eab74e5baf96311e91d70949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40669 } }
I20260808 15:53:52.919040 27529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:52.911311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42179 (local address 127.0.0.1:40669)
0808 15:53:52.911520 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:52.911527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:52.911978 (+ 451us) server_negotiation.cc:407] Connection header received
0808 15:53:52.912154 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:52.912160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:52.912239 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:52.912356 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:52.912954 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:52.914068 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:52.916013 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:52.916772 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:52.916906 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:52.918139 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:52.918147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:52.918155 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:52.918179 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:52.918250 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:52.918272 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:52.918421 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:52.918662 (+ 241us) server_negotiation.cc:299] Negotiation successful
0808 15:53:52.918723 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:53:52.918987 27525 leader_election.cc:302] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 082c2267eab74e5baf96311e91d70949; no voters:
I20260808 15:53:52.919663 27525 leader_election.cc:288] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:52.919880 27538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:52.921878 27525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:52.922982 27538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [term 1 LEADER]: Becoming Leader. State: Replica: 082c2267eab74e5baf96311e91d70949, State: Running, Role: LEADER
I20260808 15:53:52.923893 27538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082c2267eab74e5baf96311e91d70949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40669 } }
I20260808 15:53:52.928344 27541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "082c2267eab74e5baf96311e91d70949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082c2267eab74e5baf96311e91d70949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40669 } } }
I20260808 15:53:52.928788 27542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 082c2267eab74e5baf96311e91d70949. Latest consensus state: current_term: 1 leader_uuid: "082c2267eab74e5baf96311e91d70949" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "082c2267eab74e5baf96311e91d70949" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40669 } } }
I20260808 15:53:52.929086 27542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:52.928802 27541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:52.930008 27558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:52.934983 27558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:52.946033 27558 catalog_manager.cc:1347] Generated new cluster ID: 04d7673b188441c8a27f02d7bb9b951a
I20260808 15:53:52.946194 27558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:52.957116 27558 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:52.958389 27558 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:52.964023 27558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 082c2267eab74e5baf96311e91d70949: Generated new TSK 0
I20260808 15:53:52.965138 27558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:52.973747 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27425
W20260808 15:53:52.975095 27519 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54361
W20260808 15:53:52.975919 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27425: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:53:52.976115 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 27425
W20260808 15:53:53.008420 27404 connection.cc:535] client connection to 127.0.0.1:40669 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40669): Connection reset by peer (error 104)
2026-08-08T15:53:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:53 dist-test-slave-twwt krb5kdc[27569](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:53 dist-test-slave-twwt krb5kdc[27569](info): set up 2 sockets
Aug 08 15:53:53 dist-test-slave-twwt krb5kdc[27569](info): commencing operation
krb5kdc: starting...
W20260808 15:53:54.343115 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:54 dist-test-slave-twwt krb5kdc[27569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:53:54.697899 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:55.063185 27585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:55.063517 27585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:55.063613 27585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:55.074728 27585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:55.074857 27585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:55.074958 27585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:55.075062 27585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:55.075145 27585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:55.092080 27585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:55.094007 27585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:55.096020 27585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:55.113623 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:55.113729 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:55.113688 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:55.115106 27585 server_base.cc:1034] running on GCE node
I20260808 15:53:55.116446 27585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:55.118963 27585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:55.120208 27585 hybrid_clock.cc:648] HybridClock initialized: now 1786204435120140 us; error 63 us; skew 500 ppm
I20260808 15:53:55.120628 27585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:55 dist-test-slave-twwt krb5kdc[27569](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:55.125206 27585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:53:55.127297 27585 webserver.cc:460] Webserver started at http://127.0.0.1:33177/ using document root <none> and password file <none>
I20260808 15:53:55.128317 27585 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:55.128485 27585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:55.128912 27585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:55.132645 27585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "f99c941ce8524e86a0dfbae3bcafe6b3"
format_stamp: "Formatted at 2026-08-08 15:53:55 on dist-test-slave-twwt"
server_key: "a128a31ca305fe53725da6aa1c7817d9"
server_key_iv: "7275028eaa9b95e578467e92404bf7b2"
server_key_version: "encryptionkey@0"
I20260808 15:53:55.133433 27585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "f99c941ce8524e86a0dfbae3bcafe6b3"
format_stamp: "Formatted at 2026-08-08 15:53:55 on dist-test-slave-twwt"
server_key: "a128a31ca305fe53725da6aa1c7817d9"
server_key_iv: "7275028eaa9b95e578467e92404bf7b2"
server_key_version: "encryptionkey@0"
I20260808 15:53:55.138870 27585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:53:55.142484 27603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:55.143953 27585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:53:55.144142 27585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "f99c941ce8524e86a0dfbae3bcafe6b3"
format_stamp: "Formatted at 2026-08-08 15:53:55 on dist-test-slave-twwt"
server_key: "a128a31ca305fe53725da6aa1c7817d9"
server_key_iv: "7275028eaa9b95e578467e92404bf7b2"
server_key_version: "encryptionkey@0"
I20260808 15:53:55.144333 27585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:55.150403 27585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:55.154897 27585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:55.155200 27585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:55.175207 27585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40891
I20260808 15:53:55.175307 27687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40891 every 8 connection(s)
I20260808 15:53:55.176823 27585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:55.180701 27688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:55.186245 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 27585
I20260808 15:53:55.186455 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:55.186957 27987 external_mini_cluster.cc:1428] Setting key 8b028936892fd47958778c8036523df3
I20260808 15:53:55.191890 27688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3: Bootstrap starting.
I20260808 15:53:55.195246 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:55.188959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52131 (local address 127.0.0.1:40891)
0808 15:53:55.189893 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:55.189905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:55.189950 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:53:55.191678 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:55.191702 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:55.192114 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:55.192471 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:55.193596 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:55.193603 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:55.193618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:55.193633 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:55.193701 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:55.193720 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:55.193928 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:55.194028 (+ 100us) server_negotiation.cc:299] Negotiation successful
0808 15:53:55.194227 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":435,"threads_started":1}
I20260808 15:53:55.195912 27688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:55.197149 27688 log.cc:824] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:55.199397 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:53:55.200218 27688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3: No bootstrap required, opened a new log
I20260808 15:53:55.206754 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:55.203857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55737 (local address 127.0.0.1:40891)
0808 15:53:55.204097 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:55.204103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:55.204436 (+ 333us) server_negotiation.cc:407] Connection header received
0808 15:53:55.204646 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:55.204650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:55.204708 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:55.204861 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:55.206006 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:55.206012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:55.206017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:55.206037 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:55.206085 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:55.206103 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:55.206230 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:55.206353 (+ 123us) server_negotiation.cc:299] Negotiation successful
0808 15:53:55.206432 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260808 15:53:55.208524 27688 raft_consensus.cc:348] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40891 } }
I20260808 15:53:55.208905 27688 raft_consensus.cc:374] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:55.209023 27688 raft_consensus.cc:729] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f99c941ce8524e86a0dfbae3bcafe6b3, State: Initialized, Role: FOLLOWER
I20260808 15:53:55.209652 27688 consensus_queue.cc:260] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40891 } }
I20260808 15:53:55.209905 27688 raft_consensus.cc:388] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:55.210057 27688 raft_consensus.cc:482] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:55.210242 27688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:55.212615 27688 raft_consensus.cc:504] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40891 } }
I20260808 15:53:55.213131 27688 leader_election.cc:302] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f99c941ce8524e86a0dfbae3bcafe6b3; no voters:
I20260808 15:53:55.213814 27688 leader_election.cc:288] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:55.214068 27703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:55.214953 27703 raft_consensus.cc:686] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [term 1 LEADER]: Becoming Leader. State: Replica: f99c941ce8524e86a0dfbae3bcafe6b3, State: Running, Role: LEADER
I20260808 15:53:55.215478 27703 consensus_queue.cc:237] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40891 } }
I20260808 15:53:55.216141 27688 sys_catalog.cc:564] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:55.218418 27710 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f99c941ce8524e86a0dfbae3bcafe6b3. Latest consensus state: current_term: 1 leader_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40891 } } }
I20260808 15:53:55.218951 27710 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:55.219568 27708 sys_catalog.cc:455] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f99c941ce8524e86a0dfbae3bcafe6b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40891 } } }
I20260808 15:53:55.219909 27721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:55.219918 27708 sys_catalog.cc:458] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:55.232577 27721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:55.241952 27721 catalog_manager.cc:1347] Generated new cluster ID: a6740f6daf114d68a3aadb2a7900133d
I20260808 15:53:55.242082 27721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:55.270866 27721 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:55.271354 27721 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:55.280014 27721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f99c941ce8524e86a0dfbae3bcafe6b3: Generated new TSK 0
I20260808 15:53:55.280881 27721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:55.293011 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27585
W20260808 15:53:55.294195 27682 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52131
W20260808 15:53:55.295001 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:53:55.295145 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 27585
2026-08-08T15:53:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:55 dist-test-slave-twwt krb5kdc[27741](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:55 dist-test-slave-twwt krb5kdc[27741](info): set up 2 sockets
Aug 08 15:53:55 dist-test-slave-twwt krb5kdc[27741](info): commencing operation
krb5kdc: starting...
W20260808 15:53:56.659086 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.002s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:56 dist-test-slave-twwt krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:53:57.015158 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45435
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:57.373313 27765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:57.373625 27765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:57.373723 27765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:57.380999 27765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:57.381124 27765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:57.381172 27765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:57.381235 27765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:57.381279 27765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:57.391302 27765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:57.392666 27765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:57.394001 27765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:57.406183 27771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:57.406133 27776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:57.406230 27772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:57.407099 27765 server_base.cc:1034] running on GCE node
I20260808 15:53:57.407914 27765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:57.409353 27765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:57.410559 27765 hybrid_clock.cc:648] HybridClock initialized: now 1786204437410516 us; error 38 us; skew 500 ppm
I20260808 15:53:57.410974 27765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:57 dist-test-slave-twwt krb5kdc[27741](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:57.418033 27765 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:53:57.419746 27765 webserver.cc:460] Webserver started at http://127.0.0.1:34717/ using document root <none> and password file <none>
I20260808 15:53:57.420486 27765 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:57.420644 27765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:57.420974 27765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:57.424209 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "8f1c90e6d70c46699afaf0a477e23cb2"
format_stamp: "Formatted at 2026-08-08 15:53:57 on dist-test-slave-twwt"
server_key: "d1ec3b76683543fe514b32ebff2c6610"
server_key_iv: "84ad1b2c03c34503098c776322011848"
server_key_version: "encryptionkey@0"
I20260808 15:53:57.425052 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "8f1c90e6d70c46699afaf0a477e23cb2"
format_stamp: "Formatted at 2026-08-08 15:53:57 on dist-test-slave-twwt"
server_key: "d1ec3b76683543fe514b32ebff2c6610"
server_key_iv: "84ad1b2c03c34503098c776322011848"
server_key_version: "encryptionkey@0"
I20260808 15:53:57.430585 27765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260808 15:53:57.434271 27784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:57.435737 27765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260808 15:53:57.435952 27765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "8f1c90e6d70c46699afaf0a477e23cb2"
format_stamp: "Formatted at 2026-08-08 15:53:57 on dist-test-slave-twwt"
server_key: "d1ec3b76683543fe514b32ebff2c6610"
server_key_iv: "84ad1b2c03c34503098c776322011848"
server_key_version: "encryptionkey@0"
I20260808 15:53:57.436164 27765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:57.465184 27765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:57.469297 27765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:57.469620 27765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:57.489924 27765 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45435
I20260808 15:53:57.489980 27867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45435 every 8 connection(s)
I20260808 15:53:57.491490 27765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:57.495564 27868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:57.500774 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 27765
I20260808 15:53:57.500977 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:57.501435 27987 external_mini_cluster.cc:1428] Setting key fbc6115c421f69d47b6118c1d5064c3a
I20260808 15:53:57.506599 27868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2: Bootstrap starting.
I20260808 15:53:57.510878 27868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:57.512181 27868 log.cc:824] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:57.515893 27868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2: No bootstrap required, opened a new log
Aug 08 15:53:57 dist-test-slave-twwt krb5kdc[27741](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:53:57.524219 27868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 15:53:57.524621 27868 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:57.524758 27868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1c90e6d70c46699afaf0a477e23cb2, State: Initialized, Role: FOLLOWER
I20260808 15:53:57.525440 27868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 15:53:57.525712 27868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:57.525874 27868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:57.526062 27868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:57.526463 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:57.503362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39033 (local address 127.0.0.1:45435)
0808 15:53:57.504380 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:57.504392 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:57.504446 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:53:57.506305 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:57.506328 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:57.506829 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:57.507171 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:57.509268 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:57.510727 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:57.512537 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:57.513656 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:57.513812 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:57.518741 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:57.518749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:57.518766 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:57.518800 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:57.522081 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:57.522850 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:57.522857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:57.522863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:57.523054 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:57.523497 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:57.523501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:57.523503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:57.524017 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:57.524350 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:57.524814 (+ 464us) server_negotiation.cc:299] Negotiation successful
0808 15:53:57.525269 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":542,"threads_started":1}
I20260808 15:53:57.528563 27868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 15:53:57.529165 27868 leader_election.cc:302] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f1c90e6d70c46699afaf0a477e23cb2; no voters:
I20260808 15:53:57.529907 27868 leader_election.cc:288] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:57.530011 27873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:57.530346 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:53:57.530789 27873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1c90e6d70c46699afaf0a477e23cb2, State: Running, Role: LEADER
I20260808 15:53:57.531378 27873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } }
I20260808 15:53:57.533280 27873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f1c90e6d70c46699afaf0a477e23cb2. Latest consensus state: current_term: 1 leader_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } } }
I20260808 15:53:57.534754 27873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:57.535262 27876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f1c90e6d70c46699afaf0a477e23cb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45435 } } }
I20260808 15:53:57.535774 27876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:57.538154 27868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:57.538386 27881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:57.544643 27881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:57.566967 27881 catalog_manager.cc:1347] Generated new cluster ID: 16d72a9743cc42a4b09274cc30ed53e8
I20260808 15:53:57.567117 27881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:57.576768 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:57.542206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59367 (local address 127.0.0.1:45435)
0808 15:53:57.542384 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:57.542390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:57.554840 (+ 12450us) server_negotiation.cc:407] Connection header received
0808 15:53:57.555164 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:57.555176 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:57.555278 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:57.555353 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:57.558092 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:57.559645 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:57.562211 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:57.563222 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:57.563374 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:57.569524 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:57.569530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:57.569536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:53:57.569575 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:57.573386 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:57.574239 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:57.574244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:57.574248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:57.574409 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:53:57.574860 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:53:57.574865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:53:57.574868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:53:57.575235 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:57.575454 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:57.576151 (+ 697us) server_negotiation.cc:299] Negotiation successful
0808 15:53:57.576378 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:53:57.581759 27881 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:57.583600 27881 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:57.593348 27881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f1c90e6d70c46699afaf0a477e23cb2: Generated new TSK 0
I20260808 15:53:57.594515 27881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:57.601079 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27765
I20260808 15:53:57.602645 27858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:53:57.691573 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 27765
2026-08-08T15:53:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:53:57 dist-test-slave-twwt krb5kdc[27918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:53:57 dist-test-slave-twwt krb5kdc[27918](info): set up 2 sockets
Aug 08 15:53:57 dist-test-slave-twwt krb5kdc[27918](info): commencing operation
krb5kdc: starting...
W20260808 15:53:59.050690 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:53:59 dist-test-slave-twwt krb5kdc[27918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:53:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:53:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:53:59.407295 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38649
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:53:59.762706 27936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:53:59.763051 27936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:53:59.763222 27936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:53:59.770587 27936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:53:59.770759 27936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:53:59.770866 27936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:53:59.770980 27936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:53:59.771059 27936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:53:59.781344 27936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:53:59.782898 27936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:53:59.784273 27936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:53:59.801091 27942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:59.801332 27947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:53:59.802001 27945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:53:59.802790 27936 server_base.cc:1034] running on GCE node
I20260808 15:53:59.803730 27936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:53:59.805241 27936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:53:59.806484 27936 hybrid_clock.cc:648] HybridClock initialized: now 1786204439806445 us; error 66 us; skew 500 ppm
I20260808 15:53:59.806903 27936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:53:59 dist-test-slave-twwt krb5kdc[27918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:53:59.810488 27936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:53:59.812201 27936 webserver.cc:460] Webserver started at http://127.0.0.1:41747/ using document root <none> and password file <none>
I20260808 15:53:59.812971 27936 fs_manager.cc:359] Metadata directory not provided
I20260808 15:53:59.813136 27936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:53:59.813469 27936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:53:59.816794 27936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "83a293f8b35843e89121182a440b80da"
format_stamp: "Formatted at 2026-08-08 15:53:59 on dist-test-slave-twwt"
server_key: "153e66e68a799492133d3aa573152c93"
server_key_iv: "dfe2bf2ee3975c0d544a169b3a4b5e98"
server_key_version: "encryptionkey@0"
I20260808 15:53:59.817624 27936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "83a293f8b35843e89121182a440b80da"
format_stamp: "Formatted at 2026-08-08 15:53:59 on dist-test-slave-twwt"
server_key: "153e66e68a799492133d3aa573152c93"
server_key_iv: "dfe2bf2ee3975c0d544a169b3a4b5e98"
server_key_version: "encryptionkey@0"
I20260808 15:53:59.823165 27936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:53:59.826767 27960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:53:59.828130 27936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:53:59.828322 27936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "83a293f8b35843e89121182a440b80da"
format_stamp: "Formatted at 2026-08-08 15:53:59 on dist-test-slave-twwt"
server_key: "153e66e68a799492133d3aa573152c93"
server_key_iv: "dfe2bf2ee3975c0d544a169b3a4b5e98"
server_key_version: "encryptionkey@0"
I20260808 15:53:59.828532 27936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:53:59.850009 27936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:53:59.854205 27936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:53:59.854508 27936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:53:59.873878 27936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38649
I20260808 15:53:59.873919 28037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38649 every 8 connection(s)
I20260808 15:53:59.875319 27936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:53:59.879158 28038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:53:59.882781 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 27936
I20260808 15:53:59.882985 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:53:59.883584 27987 external_mini_cluster.cc:1428] Setting key 3f144ccca053beb83917108f593f06b9
I20260808 15:53:59.891976 28038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da: Bootstrap starting.
I20260808 15:53:59.895854 28038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da: Neither blocks nor log segments found. Creating new log.
I20260808 15:53:59.896997 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:59.885431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37963 (local address 127.0.0.1:38649)
0808 15:53:59.886194 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:59.886206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:59.886247 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:53:59.887971 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:59.887994 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:59.888409 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:59.888736 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:59.889386 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:59.890783 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:59.892597 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:59.893250 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:59.893385 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:59.894740 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:59.894748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:59.894762 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:59.895281 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:59.895362 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:59.895388 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:59.895476 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:59.895725 (+ 249us) server_negotiation.cc:299] Negotiation successful
0808 15:53:59.895994 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":339,"threads_started":1}
I20260808 15:53:59.896976 28038 log.cc:824] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da: Log is configured to *not* fsync() on all Append() calls
I20260808 15:53:59.900102 28038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da: No bootstrap required, opened a new log
I20260808 15:53:59.900851 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:53:59.908706 28038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a293f8b35843e89121182a440b80da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260808 15:53:59.909243 28038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:53:59.909355 28038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a293f8b35843e89121182a440b80da, State: Initialized, Role: FOLLOWER
I20260808 15:53:59.910233 28038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a293f8b35843e89121182a440b80da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260808 15:53:59.910657 28038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:53:59.910887 28038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:53:59.911089 28038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:53:59.912890 28043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:53:59.905490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50003 (local address 127.0.0.1:38649)
0808 15:53:59.905697 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:53:59.905720 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0808 15:53:59.906063 (+ 343us) server_negotiation.cc:407] Connection header received
0808 15:53:59.906227 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:53:59.906231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:53:59.906306 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:53:59.906403 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:53:59.907052 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:59.907795 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:59.909650 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:53:59.910448 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:53:59.910667 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:53:59.912104 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:53:59.912109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:53:59.912112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:53:59.912129 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:53:59.912169 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:53:59.912186 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:53:59.912260 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0808 15:53:59.912424 (+ 164us) server_negotiation.cc:299] Negotiation successful
0808 15:53:59.912493 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:53:59.914130 28038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a293f8b35843e89121182a440b80da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260808 15:53:59.914707 28038 leader_election.cc:302] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a293f8b35843e89121182a440b80da; no voters:
I20260808 15:53:59.915325 28038 leader_election.cc:288] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:53:59.915491 28055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:53:59.916322 28055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [term 1 LEADER]: Becoming Leader. State: Replica: 83a293f8b35843e89121182a440b80da, State: Running, Role: LEADER
I20260808 15:53:59.916932 28055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a293f8b35843e89121182a440b80da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } }
I20260808 15:53:59.917609 28038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:53:59.919715 28059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a293f8b35843e89121182a440b80da. Latest consensus state: current_term: 1 leader_uuid: "83a293f8b35843e89121182a440b80da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a293f8b35843e89121182a440b80da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } } }
I20260808 15:53:59.920189 28059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:59.922328 28058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a293f8b35843e89121182a440b80da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a293f8b35843e89121182a440b80da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38649 } } }
I20260808 15:53:59.922739 28067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:53:59.922771 28058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da [sys.catalog]: This master's current role is: LEADER
I20260808 15:53:59.928297 28067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:53:59.938179 28067 catalog_manager.cc:1347] Generated new cluster ID: 38f07175302640a5a4c688c50e1443bf
I20260808 15:53:59.938315 28067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:53:59.963102 28067 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:53:59.965009 28067 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:53:59.973762 28067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a293f8b35843e89121182a440b80da: Generated new TSK 0
I20260808 15:53:59.974920 28067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:53:59.987851 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27936
W20260808 15:53:59.989252 28031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37963
W20260808 15:53:59.989980 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 27936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:53:59.990142 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 27936
2026-08-08T15:54:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:00 dist-test-slave-twwt krb5kdc[28089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:00 dist-test-slave-twwt krb5kdc[28089](info): set up 2 sockets
Aug 08 15:54:00 dist-test-slave-twwt krb5kdc[28089](info): commencing operation
krb5kdc: starting...
W20260808 15:54:01.364405 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.002s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:01 dist-test-slave-twwt krb5kdc[28089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:01.720332 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:02.087137 28115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:02.087446 28115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:02.087656 28115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:02.095278 28115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:02.095404 28115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:02.095525 28115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:02.095636 28115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:02.095705 28115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:02.106065 28115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:02.107537 28115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:02.108973 28115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:02.121141 28122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:02.121214 28126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:02.121237 28123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:02.122241 28115 server_base.cc:1034] running on GCE node
I20260808 15:54:02.123087 28115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:02.124496 28115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:02.125756 28115 hybrid_clock.cc:648] HybridClock initialized: now 1786204442125711 us; error 35 us; skew 500 ppm
I20260808 15:54:02.126127 28115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:02 dist-test-slave-twwt krb5kdc[28089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:02.129390 28115 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:02.131058 28115 webserver.cc:460] Webserver started at http://127.0.0.1:34385/ using document root <none> and password file <none>
I20260808 15:54:02.131798 28115 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:02.131963 28115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:02.132293 28115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:02.135648 28115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "dda3901756ba446f85a263c7343de072"
format_stamp: "Formatted at 2026-08-08 15:54:02 on dist-test-slave-twwt"
server_key: "8fc612d52502c0010a4d80bbb9256629"
server_key_iv: "903bf44ef89cf482c100012f40444f34"
server_key_version: "encryptionkey@0"
I20260808 15:54:02.136547 28115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "dda3901756ba446f85a263c7343de072"
format_stamp: "Formatted at 2026-08-08 15:54:02 on dist-test-slave-twwt"
server_key: "8fc612d52502c0010a4d80bbb9256629"
server_key_iv: "903bf44ef89cf482c100012f40444f34"
server_key_version: "encryptionkey@0"
I20260808 15:54:02.142041 28115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:54:02.145706 28138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:02.147281 28115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:02.147490 28115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "dda3901756ba446f85a263c7343de072"
format_stamp: "Formatted at 2026-08-08 15:54:02 on dist-test-slave-twwt"
server_key: "8fc612d52502c0010a4d80bbb9256629"
server_key_iv: "903bf44ef89cf482c100012f40444f34"
server_key_version: "encryptionkey@0"
I20260808 15:54:02.147709 28115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:02.153806 28115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:02.158352 28115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:02.158741 28115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:02.178125 28115 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38289
I20260808 15:54:02.178171 28218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38289 every 8 connection(s)
I20260808 15:54:02.179659 28115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:02.183552 28219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:02.186777 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28115
I20260808 15:54:02.187024 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:02.187577 27987 external_mini_cluster.cc:1428] Setting key a5ec38ff0f28ea2b2067aa91930f4c03
I20260808 15:54:02.195788 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:02.189613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52643 (local address 127.0.0.1:38289)
0808 15:54:02.190401 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:02.190414 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:02.190455 (+ 41us) server_negotiation.cc:407] Connection header received
0808 15:54:02.192179 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:02.192202 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:02.192559 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:02.193012 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:02.194134 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:02.194140 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:02.194153 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:02.194166 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:02.194232 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:02.194250 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:02.194485 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:02.194655 (+ 170us) server_negotiation.cc:299] Negotiation successful
0808 15:54:02.194863 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":345,"threads_started":1}
I20260808 15:54:02.196059 28219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072: Bootstrap starting.
I20260808 15:54:02.200150 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:02.200254 28219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:02.201357 28219 log.cc:824] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:02.204445 28219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072: No bootstrap required, opened a new log
I20260808 15:54:02.207522 28224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:02.204747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37783 (local address 127.0.0.1:38289)
0808 15:54:02.204992 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:02.204998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:02.205508 (+ 510us) server_negotiation.cc:407] Connection header received
0808 15:54:02.205641 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:02.205646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:02.205716 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:02.205834 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:02.206827 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:02.206837 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:02.206842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:02.206862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:02.206918 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:02.206942 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:02.207080 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:02.207155 (+ 75us) server_negotiation.cc:299] Negotiation successful
0808 15:54:02.207223 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:54:02.214043 28219 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda3901756ba446f85a263c7343de072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260808 15:54:02.214457 28219 raft_consensus.cc:374] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:02.214537 28219 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda3901756ba446f85a263c7343de072, State: Initialized, Role: FOLLOWER
I20260808 15:54:02.215219 28219 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda3901756ba446f85a263c7343de072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260808 15:54:02.215441 28219 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:02.215551 28219 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:02.215682 28219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:02.218083 28219 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda3901756ba446f85a263c7343de072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260808 15:54:02.218711 28219 leader_election.cc:302] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda3901756ba446f85a263c7343de072; no voters:
I20260808 15:54:02.219319 28219 leader_election.cc:288] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:02.219491 28236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:02.220216 28236 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [term 1 LEADER]: Becoming Leader. State: Replica: dda3901756ba446f85a263c7343de072, State: Running, Role: LEADER
I20260808 15:54:02.220748 28236 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda3901756ba446f85a263c7343de072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } }
I20260808 15:54:02.221635 28219 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:02.224057 28237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dda3901756ba446f85a263c7343de072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda3901756ba446f85a263c7343de072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } } }
I20260808 15:54:02.223932 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda3901756ba446f85a263c7343de072. Latest consensus state: current_term: 1 leader_uuid: "dda3901756ba446f85a263c7343de072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda3901756ba446f85a263c7343de072" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38289 } } }
I20260808 15:54:02.224567 28237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:02.224610 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:02.226449 28248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:02.232296 28248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:02.242100 28248 catalog_manager.cc:1347] Generated new cluster ID: d032bfc3734240cf815de078d27bd192
I20260808 15:54:02.242254 28248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:02.261584 28248 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:02.262208 28248 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:02.270298 28248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dda3901756ba446f85a263c7343de072: Generated new TSK 0
I20260808 15:54:02.271209 28248 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:02.279258 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28115
W20260808 15:54:02.280673 28208 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52643
W20260808 15:54:02.281457 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28115: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:02.281631 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28115
2026-08-08T15:54:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:02 dist-test-slave-twwt krb5kdc[28270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:02 dist-test-slave-twwt krb5kdc[28270](info): set up 2 sockets
Aug 08 15:54:02 dist-test-slave-twwt krb5kdc[28270](info): commencing operation
krb5kdc: starting...
W20260808 15:54:03.639127 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.005s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:03 dist-test-slave-twwt krb5kdc[28270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:04.000227 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:04.361802 28286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:04.362181 28286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:04.362361 28286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:04.370106 28286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:04.370292 28286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:04.370416 28286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:04.370532 28286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:04.370611 28286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:04.381184 28286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:04.382743 28286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:04.384202 28286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:04.396973 28297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:04.396977 28295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:04.397045 28294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:04.397914 28286 server_base.cc:1034] running on GCE node
I20260808 15:54:04.398805 28286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:04.400318 28286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:04.401564 28286 hybrid_clock.cc:648] HybridClock initialized: now 1786204444401515 us; error 42 us; skew 500 ppm
I20260808 15:54:04.401921 28286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:04 dist-test-slave-twwt krb5kdc[28270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:04.413363 28286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:04.415494 28286 webserver.cc:460] Webserver started at http://127.0.0.1:41499/ using document root <none> and password file <none>
I20260808 15:54:04.416389 28286 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:04.416558 28286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:04.416895 28286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:04.420244 28286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "6cffc63fc6af48158daa7a852c2f3673"
format_stamp: "Formatted at 2026-08-08 15:54:04 on dist-test-slave-twwt"
server_key: "8bd2769dd4a7604a0e29a12e9bd2314b"
server_key_iv: "3e6dd7fe5d28e5477f18c33371d47eb8"
server_key_version: "encryptionkey@0"
I20260808 15:54:04.421078 28286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "6cffc63fc6af48158daa7a852c2f3673"
format_stamp: "Formatted at 2026-08-08 15:54:04 on dist-test-slave-twwt"
server_key: "8bd2769dd4a7604a0e29a12e9bd2314b"
server_key_iv: "3e6dd7fe5d28e5477f18c33371d47eb8"
server_key_version: "encryptionkey@0"
I20260808 15:54:04.426746 28286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260808 15:54:04.430786 28308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:04.432300 28286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:04.432518 28286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "6cffc63fc6af48158daa7a852c2f3673"
format_stamp: "Formatted at 2026-08-08 15:54:04 on dist-test-slave-twwt"
server_key: "8bd2769dd4a7604a0e29a12e9bd2314b"
server_key_iv: "3e6dd7fe5d28e5477f18c33371d47eb8"
server_key_version: "encryptionkey@0"
I20260808 15:54:04.432767 28286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:04.455958 28286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:04.460434 28286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:04.460764 28286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:04.479812 28286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36363
I20260808 15:54:04.479833 28385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36363 every 8 connection(s)
I20260808 15:54:04.481302 28286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:04.485217 28387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:04.487641 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28286
I20260808 15:54:04.487831 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:04.488325 27987 external_mini_cluster.cc:1428] Setting key a1f85cb7fe8d4a6024038b04b1f81b61
I20260808 15:54:04.497876 28387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673: Bootstrap starting.
I20260808 15:54:04.501981 28387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:04.503350 28387 log.cc:824] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:54:04 dist-test-slave-twwt krb5kdc[28270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:54:04.506557 28387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673: No bootstrap required, opened a new log
I20260808 15:54:04.512789 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:04.490259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:40335 (local address 127.0.0.1:36363)
0808 15:54:04.491185 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:04.491197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:04.491246 (+ 49us) server_negotiation.cc:407] Connection header received
0808 15:54:04.492926 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:04.492950 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:04.493367 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:04.493713 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:04.495711 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:04.497060 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:04.498906 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:04.499621 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:04.499750 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:04.504787 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:04.504795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:04.504810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:04.504847 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:04.508089 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:04.509023 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:04.509030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:04.509040 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:04.509219 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:04.509745 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:04.509750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:04.509753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:04.510391 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:04.510854 (+ 463us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:04.511295 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 15:54:04.511648 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":441,"threads_started":1}
I20260808 15:54:04.514976 28387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cffc63fc6af48158daa7a852c2f3673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260808 15:54:04.515388 28387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:04.515529 28387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cffc63fc6af48158daa7a852c2f3673, State: Initialized, Role: FOLLOWER
I20260808 15:54:04.516266 28387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cffc63fc6af48158daa7a852c2f3673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260808 15:54:04.516580 28387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:04.516747 28387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:04.516954 28387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:04.517228 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:04.520354 28387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cffc63fc6af48158daa7a852c2f3673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260808 15:54:04.521049 28387 leader_election.cc:302] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cffc63fc6af48158daa7a852c2f3673; no voters:
I20260808 15:54:04.521814 28387 leader_election.cc:288] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:04.521956 28399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:04.524348 28387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:04.528925 28399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [term 1 LEADER]: Becoming Leader. State: Replica: 6cffc63fc6af48158daa7a852c2f3673, State: Running, Role: LEADER
I20260808 15:54:04.529697 28399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cffc63fc6af48158daa7a852c2f3673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } }
I20260808 15:54:04.532888 28403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cffc63fc6af48158daa7a852c2f3673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cffc63fc6af48158daa7a852c2f3673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } } }
I20260808 15:54:04.533282 28403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:04.533780 28405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cffc63fc6af48158daa7a852c2f3673. Latest consensus state: current_term: 1 leader_uuid: "6cffc63fc6af48158daa7a852c2f3673" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cffc63fc6af48158daa7a852c2f3673" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36363 } } }
I20260808 15:54:04.534083 28405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:04.542870 28414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:04.544703 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:04.522894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:49599 (local address 127.0.0.1:36363)
0808 15:54:04.523174 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:04.523184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:04.524270 (+ 1086us) server_negotiation.cc:407] Connection header received
0808 15:54:04.524650 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:04.524656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:04.524741 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:04.524861 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:04.527361 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:04.528907 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:04.531225 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:04.532013 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:04.532247 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:04.536628 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:04.536639 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:04.536647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:04.536702 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:04.540927 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:04.541789 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:04.541794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:04.541797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:04.541970 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:04.542677 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:04.542684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:04.542690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:04.543186 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:04.543771 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:04.544152 (+ 381us) server_negotiation.cc:299] Negotiation successful
0808 15:54:04.544354 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":95}
I20260808 15:54:04.548623 28414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:04.558846 28414 catalog_manager.cc:1347] Generated new cluster ID: b16d45f8d3b642e2a8b45ec169005e9f
I20260808 15:54:04.558983 28414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:04.592775 28414 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:04.594726 28414 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:04.603273 28414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cffc63fc6af48158daa7a852c2f3673: Generated new TSK 0
I20260808 15:54:04.604387 28414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:04.620221 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28286
I20260808 15:54:04.622139 28377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:04.720430 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28286
W20260808 15:54:04.750006 28268 connection.cc:535] client connection to 127.0.0.1:36363 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36363): Connection reset by peer (error 104)
2026-08-08T15:54:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:04 dist-test-slave-twwt krb5kdc[28431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:04 dist-test-slave-twwt krb5kdc[28431](info): set up 2 sockets
Aug 08 15:54:04 dist-test-slave-twwt krb5kdc[28431](info): commencing operation
krb5kdc: starting...
W20260808 15:54:06.098969 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.007s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:06 dist-test-slave-twwt krb5kdc[28431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:06.470049 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:06.834790 28449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:06.835088 28449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:06.835182 28449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:06.842224 28449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:06.842337 28449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:06.842384 28449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:06.842453 28449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:06.842496 28449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:06.852303 28449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:06.853571 28449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:06.854871 28449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:06.867169 28455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:06.867199 28464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:06.867395 28458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:06.868083 28449 server_base.cc:1034] running on GCE node
I20260808 15:54:06.868956 28449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:06.870481 28449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:06.871742 28449 hybrid_clock.cc:648] HybridClock initialized: now 1786204446871697 us; error 36 us; skew 500 ppm
I20260808 15:54:06.872169 28449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:06 dist-test-slave-twwt krb5kdc[28431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:06.875651 28449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:06.877240 28449 webserver.cc:460] Webserver started at http://127.0.0.1:33821/ using document root <none> and password file <none>
I20260808 15:54:06.877945 28449 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:06.878051 28449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:06.878315 28449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:06.881578 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "3f30163d71af45cf9734dea73042576e"
format_stamp: "Formatted at 2026-08-08 15:54:06 on dist-test-slave-twwt"
server_key: "8d70d8f73659930dfb0ccb8b029048a3"
server_key_iv: "35f0c1f37bd416a68c2ca93f99fb6817"
server_key_version: "encryptionkey@0"
I20260808 15:54:06.882352 28449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "3f30163d71af45cf9734dea73042576e"
format_stamp: "Formatted at 2026-08-08 15:54:06 on dist-test-slave-twwt"
server_key: "8d70d8f73659930dfb0ccb8b029048a3"
server_key_iv: "35f0c1f37bd416a68c2ca93f99fb6817"
server_key_version: "encryptionkey@0"
I20260808 15:54:06.888255 28449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260808 15:54:06.892033 28473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:06.893565 28449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:54:06.893788 28449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "3f30163d71af45cf9734dea73042576e"
format_stamp: "Formatted at 2026-08-08 15:54:06 on dist-test-slave-twwt"
server_key: "8d70d8f73659930dfb0ccb8b029048a3"
server_key_iv: "35f0c1f37bd416a68c2ca93f99fb6817"
server_key_version: "encryptionkey@0"
I20260808 15:54:06.894025 28449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:06.921188 28449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:06.925774 28449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:06.926085 28449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:06.954392 28449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42603
I20260808 15:54:06.954437 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42603 every 8 connection(s)
I20260808 15:54:06.955924 28449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:06.959384 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28449
I20260808 15:54:06.959569 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:06.960013 28565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:06.960052 27987 external_mini_cluster.cc:1428] Setting key a75af2dd1c73b927d126e1a128ba6289
I20260808 15:54:06.975929 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:06.962016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:57529 (local address 127.0.0.1:42603)
0808 15:54:06.962949 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:06.962964 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:06.963021 (+ 57us) server_negotiation.cc:407] Connection header received
0808 15:54:06.965257 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:06.965288 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:06.965862 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:06.966351 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:06.966985 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:06.969063 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:06.970417 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:06.971533 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:06.971725 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:06.972951 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:06.972962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:06.972981 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:06.973768 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:06.973866 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:06.973901 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:06.974057 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:06.974332 (+ 275us) server_negotiation.cc:299] Negotiation successful
0808 15:54:06.974520 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":314,"threads_started":1}
I20260808 15:54:06.976081 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e: Bootstrap starting.
I20260808 15:54:06.979461 28565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:06.980060 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:06.980762 28565 log.cc:824] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:06.984413 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e: No bootstrap required, opened a new log
I20260808 15:54:06.992632 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:06.985307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:35527 (local address 127.0.0.1:42603)
0808 15:54:06.985528 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:06.985537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:06.986060 (+ 523us) server_negotiation.cc:407] Connection header received
0808 15:54:06.986293 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:06.986296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:06.986367 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:06.986465 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:06.987103 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:06.987918 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:06.989898 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:06.990404 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:06.990501 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:06.991772 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:06.991776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:06.991779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:06.991798 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:06.991835 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:06.991852 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:06.991946 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:06.992188 (+ 242us) server_negotiation.cc:299] Negotiation successful
0808 15:54:06.992264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260808 15:54:06.995090 28565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f30163d71af45cf9734dea73042576e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42603 } }
I20260808 15:54:06.995500 28565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:06.995600 28565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f30163d71af45cf9734dea73042576e, State: Initialized, Role: FOLLOWER
I20260808 15:54:06.996333 28565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f30163d71af45cf9734dea73042576e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42603 } }
I20260808 15:54:06.996647 28565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:06.996820 28565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:06.997015 28565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:06.999524 28565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f30163d71af45cf9734dea73042576e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42603 } }
I20260808 15:54:07.000079 28565 leader_election.cc:302] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f30163d71af45cf9734dea73042576e; no voters:
I20260808 15:54:07.000701 28565 leader_election.cc:288] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:07.000905 28580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:07.002143 28580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [term 1 LEADER]: Becoming Leader. State: Replica: 3f30163d71af45cf9734dea73042576e, State: Running, Role: LEADER
I20260808 15:54:07.002817 28580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f30163d71af45cf9734dea73042576e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42603 } }
I20260808 15:54:07.003237 28565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:07.005930 28583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f30163d71af45cf9734dea73042576e. Latest consensus state: current_term: 1 leader_uuid: "3f30163d71af45cf9734dea73042576e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f30163d71af45cf9734dea73042576e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42603 } } }
I20260808 15:54:07.006006 28582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f30163d71af45cf9734dea73042576e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f30163d71af45cf9734dea73042576e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42603 } } }
I20260808 15:54:07.006422 28583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:07.006474 28582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e [sys.catalog]: This master's current role is: LEADER
W20260808 15:54:07.016763 28601 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:54:07.016906 28601 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:54:07.017062 28591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:07.019398 28591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:07.028667 28591 catalog_manager.cc:1347] Generated new cluster ID: dbf47274ded04cee99ee9f9b48a6ce64
I20260808 15:54:07.028803 28591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:07.063906 28591 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:07.065240 28591 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:07.082201 28591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f30163d71af45cf9734dea73042576e: Generated new TSK 0
I20260808 15:54:07.083288 28591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:07.100440 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28449
W20260808 15:54:07.101692 28556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:57529
W20260808 15:54:07.102484 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28449: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:07.102679 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28449
W20260808 15:54:07.134270 28429 connection.cc:535] client connection to 127.0.0.1:42603 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42603): Connection reset by peer (error 104)
2026-08-08T15:54:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:07 dist-test-slave-twwt krb5kdc[28608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:07 dist-test-slave-twwt krb5kdc[28608](info): set up 2 sockets
Aug 08 15:54:07 dist-test-slave-twwt krb5kdc[28608](info): commencing operation
krb5kdc: starting...
W20260808 15:54:08.470559 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:08 dist-test-slave-twwt krb5kdc[28608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:08.829730 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:09.191197 28625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:09.191530 28625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:09.191738 28625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:09.199107 28625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:09.199254 28625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:09.199344 28625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:09.199468 28625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:09.199549 28625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:09.210258 28625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:09.211752 28625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:09.213203 28625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:09.225626 28632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:09.225852 28631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:09.225867 28636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:09.226939 28625 server_base.cc:1034] running on GCE node
I20260808 15:54:09.227748 28625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:09.229223 28625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:09.230470 28625 hybrid_clock.cc:648] HybridClock initialized: now 1786204449230416 us; error 45 us; skew 500 ppm
I20260808 15:54:09.230896 28625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:09 dist-test-slave-twwt krb5kdc[28608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:09.234362 28625 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:09.235832 28625 webserver.cc:460] Webserver started at http://127.0.0.1:40907/ using document root <none> and password file <none>
I20260808 15:54:09.236480 28625 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:09.236588 28625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:09.236868 28625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:09.239998 28625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "7a3b3bdaee964b87b00866c85e0e3a63"
format_stamp: "Formatted at 2026-08-08 15:54:09 on dist-test-slave-twwt"
server_key: "b6451513be690c3bd70f7178e6ca689a"
server_key_iv: "5b4d2788c5052ed0ffd7d45d016f48fa"
server_key_version: "encryptionkey@0"
I20260808 15:54:09.240793 28625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "7a3b3bdaee964b87b00866c85e0e3a63"
format_stamp: "Formatted at 2026-08-08 15:54:09 on dist-test-slave-twwt"
server_key: "b6451513be690c3bd70f7178e6ca689a"
server_key_iv: "5b4d2788c5052ed0ffd7d45d016f48fa"
server_key_version: "encryptionkey@0"
I20260808 15:54:09.246358 28625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260808 15:54:09.250098 28652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:09.251678 28625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:09.251844 28625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "7a3b3bdaee964b87b00866c85e0e3a63"
format_stamp: "Formatted at 2026-08-08 15:54:09 on dist-test-slave-twwt"
server_key: "b6451513be690c3bd70f7178e6ca689a"
server_key_iv: "5b4d2788c5052ed0ffd7d45d016f48fa"
server_key_version: "encryptionkey@0"
I20260808 15:54:09.252041 28625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:09.258190 28625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:09.262812 28625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:09.263089 28625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:09.281826 28625 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45533
I20260808 15:54:09.281867 28722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45533 every 8 connection(s)
I20260808 15:54:09.283376 28625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:09.284778 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28625
I20260808 15:54:09.284992 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:09.285475 27987 external_mini_cluster.cc:1428] Setting key 9c6f3f3994432611fd255b52cce042b0
I20260808 15:54:09.288235 28723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:09.294775 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:09.287407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:57289 (local address 127.0.0.1:45533)
0808 15:54:09.288268 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:09.288287 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:09.288351 (+ 64us) server_negotiation.cc:407] Connection header received
0808 15:54:09.290715 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:09.290743 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:09.291193 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:09.291667 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:09.292682 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:09.292694 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:09.292732 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:09.292757 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:09.292867 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:09.292908 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:09.293139 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:09.293269 (+ 130us) server_negotiation.cc:299] Negotiation successful
0808 15:54:09.293473 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":335,"threads_started":1}
I20260808 15:54:09.298714 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:09.303081 28723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63: Bootstrap starting.
I20260808 15:54:09.306885 28723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:09.306891 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:09.304283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:40611 (local address 127.0.0.1:45533)
0808 15:54:09.304468 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:09.304476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:09.304917 (+ 441us) server_negotiation.cc:407] Connection header received
0808 15:54:09.305067 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:09.305073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:09.305136 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:09.305252 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:09.306151 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:09.306156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:09.306161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:09.306179 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:09.306223 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:09.306241 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:09.306356 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:09.306455 (+ 99us) server_negotiation.cc:299] Negotiation successful
0808 15:54:09.306513 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260808 15:54:09.308336 28723 log.cc:824] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:09.311378 28723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63: No bootstrap required, opened a new log
I20260808 15:54:09.318773 28723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20260808 15:54:09.319157 28723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:09.319265 28723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3b3bdaee964b87b00866c85e0e3a63, State: Initialized, Role: FOLLOWER
I20260808 15:54:09.319905 28723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20260808 15:54:09.320152 28723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:09.320307 28723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:09.320484 28723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:09.322815 28723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20260808 15:54:09.323316 28723 leader_election.cc:302] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3b3bdaee964b87b00866c85e0e3a63; no voters:
I20260808 15:54:09.323998 28723 leader_election.cc:288] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:09.324182 28738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:09.325205 28738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [term 1 LEADER]: Becoming Leader. State: Replica: 7a3b3bdaee964b87b00866c85e0e3a63, State: Running, Role: LEADER
I20260808 15:54:09.325757 28738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } }
I20260808 15:54:09.326426 28723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:09.328739 28739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } } }
I20260808 15:54:09.329116 28739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:09.330080 28747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:09.329874 28740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3b3bdaee964b87b00866c85e0e3a63. Latest consensus state: current_term: 1 leader_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3b3bdaee964b87b00866c85e0e3a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45533 } } }
I20260808 15:54:09.330304 28740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:09.335559 28747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:09.344455 28747 catalog_manager.cc:1347] Generated new cluster ID: 58e6db953d354dc3bd12ab2e6c7f6d47
I20260808 15:54:09.344587 28747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:09.368412 28747 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:09.368954 28747 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:09.379866 28747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a3b3bdaee964b87b00866c85e0e3a63: Generated new TSK 0
I20260808 15:54:09.380988 28747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:09.388226 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28625
W20260808 15:54:09.389348 28718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:57289
W20260808 15:54:09.390090 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28625: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:09.390244 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28625
2026-08-08T15:54:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2284 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:09 dist-test-slave-twwt krb5kdc[28765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:09 dist-test-slave-twwt krb5kdc[28765](info): set up 2 sockets
Aug 08 15:54:09 dist-test-slave-twwt krb5kdc[28765](info): commencing operation
krb5kdc: starting...
W20260808 15:54:10.749590 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:10 dist-test-slave-twwt krb5kdc[28765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:11.113581 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:11.470386 28784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:11.470803 28784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:11.470980 28784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:11.478473 28784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:11.478686 28784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:11.478807 28784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:11.478935 28784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:11.479024 28784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:11.489118 28784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37891
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:11.490571 28784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:11.492128 28784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:11.504846 28796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:11.504971 28790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:11.504946 28792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:11.506157 28784 server_base.cc:1034] running on GCE node
I20260808 15:54:11.507045 28784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:11.508527 28784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:11.509740 28784 hybrid_clock.cc:648] HybridClock initialized: now 1786204451509693 us; error 34 us; skew 500 ppm
I20260808 15:54:11.510135 28784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:11 dist-test-slave-twwt krb5kdc[28765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:11.513525 28784 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:11.515205 28784 webserver.cc:460] Webserver started at http://127.0.0.1:38221/ using document root <none> and password file <none>
I20260808 15:54:11.515892 28784 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:11.515996 28784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:11.516250 28784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:11.519435 28784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614"
format_stamp: "Formatted at 2026-08-08 15:54:11 on dist-test-slave-twwt"
server_key: "d020a3a9c386a4357bc89eb1411a3069"
server_key_iv: "6cc8c31e949ace22766139c682034452"
server_key_version: "encryptionkey@0"
I20260808 15:54:11.520191 28784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614"
format_stamp: "Formatted at 2026-08-08 15:54:11 on dist-test-slave-twwt"
server_key: "d020a3a9c386a4357bc89eb1411a3069"
server_key_iv: "6cc8c31e949ace22766139c682034452"
server_key_version: "encryptionkey@0"
I20260808 15:54:11.525702 28784 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:11.529731 28808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:11.531291 28784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:11.531481 28784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614"
format_stamp: "Formatted at 2026-08-08 15:54:11 on dist-test-slave-twwt"
server_key: "d020a3a9c386a4357bc89eb1411a3069"
server_key_iv: "6cc8c31e949ace22766139c682034452"
server_key_version: "encryptionkey@0"
I20260808 15:54:11.531685 28784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:11.558264 28784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:11.563110 28784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:11.563437 28784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:11.583168 28784 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37891
I20260808 15:54:11.583192 28885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37891 every 8 connection(s)
I20260808 15:54:11.584695 28784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:11.588709 28887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:11.589964 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28784
I20260808 15:54:11.590163 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:11.590785 27987 external_mini_cluster.cc:1428] Setting key fa0a8983e9ac8e1f51e2b49b6b301a43
I20260808 15:54:11.603555 28887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614: Bootstrap starting.
Aug 08 15:54:11 dist-test-slave-twwt krb5kdc[28765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260808 15:54:11.607342 28887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:11.608621 28887 log.cc:824] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:11.611625 28887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614: No bootstrap required, opened a new log
I20260808 15:54:11.614883 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:11.592694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:49693 (local address 127.0.0.1:37891)
0808 15:54:11.593572 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:11.593584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:11.593632 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:54:11.595344 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:11.595371 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:11.595793 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:11.596138 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:11.597944 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:11.599318 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:11.601175 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:11.601826 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:11.601951 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:11.607013 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:11.607020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:11.607034 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:11.607076 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:11.610254 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:11.611190 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:11.611196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:11.611203 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:11.611375 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:11.611909 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:11.611921 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:11.611924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:11.612437 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:11.612806 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:11.613343 (+ 537us) server_negotiation.cc:299] Negotiation successful
0808 15:54:11.613667 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":378,"threads_started":1}
I20260808 15:54:11.619165 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:11.620637 28887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260808 15:54:11.621035 28887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:11.621150 28887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbf2a21fb8748d8a5a5e61f4f5a1614, State: Initialized, Role: FOLLOWER
I20260808 15:54:11.621762 28887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260808 15:54:11.622074 28887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:11.622236 28887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:11.622418 28887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:11.625317 28887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260808 15:54:11.625849 28887 leader_election.cc:302] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cbf2a21fb8748d8a5a5e61f4f5a1614; no voters:
I20260808 15:54:11.626480 28887 leader_election.cc:288] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:11.626657 28899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:11.627560 28899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [term 1 LEADER]: Becoming Leader. State: Replica: 9cbf2a21fb8748d8a5a5e61f4f5a1614, State: Running, Role: LEADER
I20260808 15:54:11.628194 28899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } }
I20260808 15:54:11.629398 28887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:11.631402 28906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cbf2a21fb8748d8a5a5e61f4f5a1614. Latest consensus state: current_term: 1 leader_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } } }
I20260808 15:54:11.631409 28902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbf2a21fb8748d8a5a5e61f4f5a1614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37891 } } }
I20260808 15:54:11.632040 28906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:11.632037 28902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:11.644465 28912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:11.645072 28890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:11.623828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:48677 (local address 127.0.0.1:37891)
0808 15:54:11.624006 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:11.624010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:11.624461 (+ 451us) server_negotiation.cc:407] Connection header received
0808 15:54:11.624690 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:11.624694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:11.624756 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:11.624854 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:11.626691 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:11.628152 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:11.629605 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:11.630292 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:11.630465 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:11.634482 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:11.634491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:11.634499 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:11.634553 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:11.639644 (+ 5091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:11.641039 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:11.641047 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:11.641054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:11.641264 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:11.642035 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:11.642043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:11.642049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:11.642652 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:11.644294 (+ 1642us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:11.644498 (+ 204us) server_negotiation.cc:299] Negotiation successful
0808 15:54:11.644713 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62}
I20260808 15:54:11.652118 28912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:11.661589 28912 catalog_manager.cc:1347] Generated new cluster ID: ead55a0600d84de6be7e2690834f02af
I20260808 15:54:11.661715 28912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:11.689083 28912 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:11.691009 28912 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:11.702353 28912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cbf2a21fb8748d8a5a5e61f4f5a1614: Generated new TSK 0
I20260808 15:54:11.703442 28912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:11.709345 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28784
I20260808 15:54:11.710683 28878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:11.805969 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28784
W20260808 15:54:11.832768 28763 connection.cc:535] client connection to 127.0.0.1:37891 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37891): Connection reset by peer (error 104)
2026-08-08T15:54:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:11 dist-test-slave-twwt krb5kdc[28938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:11 dist-test-slave-twwt krb5kdc[28938](info): set up 2 sockets
Aug 08 15:54:11 dist-test-slave-twwt krb5kdc[28938](info): commencing operation
krb5kdc: starting...
W20260808 15:54:13.156519 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:13 dist-test-slave-twwt krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:13.514194 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:13.870832 28964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:13.871177 28964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:13.871356 28964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:13.878546 28964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:13.878744 28964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:13.878858 28964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:13.878973 28964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:13.879065 28964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:13.889019 28964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:13.890511 28964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:13.892014 28964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:13.904258 28970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:13.904241 28972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:13.904501 28976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:13.905567 28964 server_base.cc:1034] running on GCE node
I20260808 15:54:13.906388 28964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:13.907900 28964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:13.909133 28964 hybrid_clock.cc:648] HybridClock initialized: now 1786204453909101 us; error 46 us; skew 500 ppm
I20260808 15:54:13.909497 28964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:13 dist-test-slave-twwt krb5kdc[28938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:13.912756 28964 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:13.914228 28964 webserver.cc:460] Webserver started at http://127.0.0.1:44007/ using document root <none> and password file <none>
I20260808 15:54:13.914983 28964 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:13.915092 28964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:13.915354 28964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:13.918453 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "1256088a95854251986461569a79f7d4"
format_stamp: "Formatted at 2026-08-08 15:54:13 on dist-test-slave-twwt"
server_key: "54e8b76f0173cf9cfb9d09fd9387a633"
server_key_iv: "ce73138f50b91d4bfb4257df3903691c"
server_key_version: "encryptionkey@0"
I20260808 15:54:13.919273 28964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "1256088a95854251986461569a79f7d4"
format_stamp: "Formatted at 2026-08-08 15:54:13 on dist-test-slave-twwt"
server_key: "54e8b76f0173cf9cfb9d09fd9387a633"
server_key_iv: "ce73138f50b91d4bfb4257df3903691c"
server_key_version: "encryptionkey@0"
I20260808 15:54:13.924567 28964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:54:13.928074 28985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:13.929579 28964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:54:13.929773 28964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "1256088a95854251986461569a79f7d4"
format_stamp: "Formatted at 2026-08-08 15:54:13 on dist-test-slave-twwt"
server_key: "54e8b76f0173cf9cfb9d09fd9387a633"
server_key_iv: "ce73138f50b91d4bfb4257df3903691c"
server_key_version: "encryptionkey@0"
I20260808 15:54:13.929976 28964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:13.946867 28964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:13.951819 28964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:13.952167 28964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:13.975461 28964 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40713
I20260808 15:54:13.975495 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40713 every 8 connection(s)
I20260808 15:54:13.976965 28964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:13.979809 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 28964
I20260808 15:54:13.980019 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:13.980525 27987 external_mini_cluster.cc:1428] Setting key 7ec29d452b59e5b6d1b723d7b9ad8c19
I20260808 15:54:13.981421 29064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:13.996604 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:13.982594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:55157 (local address 127.0.0.1:40713)
0808 15:54:13.983510 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:13.983527 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:13.983582 (+ 55us) server_negotiation.cc:407] Connection header received
0808 15:54:13.985403 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:13.985435 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:13.986031 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:13.986525 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:13.987190 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:13.989329 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:13.990912 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:13.991962 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:13.992152 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:13.993503 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:13.993514 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:13.993534 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:13.994350 (+ 816us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:13.994451 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:13.994490 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:13.994687 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:13.995025 (+ 338us) server_negotiation.cc:299] Negotiation successful
0808 15:54:13.995228 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":325,"threads_started":1}
I20260808 15:54:13.997865 29064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4: Bootstrap starting.
I20260808 15:54:14.000988 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:14.001633 29064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:14.002772 29064 log.cc:824] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:14.006371 29064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4: No bootstrap required, opened a new log
I20260808 15:54:14.012503 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:14.005882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:45735 (local address 127.0.0.1:40713)
0808 15:54:14.006075 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:14.006084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:14.006488 (+ 404us) server_negotiation.cc:407] Connection header received
0808 15:54:14.006751 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:14.006755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:14.006820 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:14.006949 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:14.007508 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:14.008336 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:14.009680 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:14.010148 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:14.010296 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:14.011689 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:14.011694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:14.011700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:14.011720 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:14.011764 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:14.011781 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:14.011936 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:14.012146 (+ 210us) server_negotiation.cc:299] Negotiation successful
0808 15:54:14.012209 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260808 15:54:14.015149 29064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256088a95854251986461569a79f7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260808 15:54:14.015676 29064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:14.015816 29064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256088a95854251986461569a79f7d4, State: Initialized, Role: FOLLOWER
I20260808 15:54:14.016489 29064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256088a95854251986461569a79f7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260808 15:54:14.016729 29064 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:14.016908 29064 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:14.017079 29064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:14.019451 29064 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256088a95854251986461569a79f7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260808 15:54:14.020052 29064 leader_election.cc:302] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1256088a95854251986461569a79f7d4; no voters:
I20260808 15:54:14.020789 29064 leader_election.cc:288] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:14.020908 29086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:14.021646 29086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [term 1 LEADER]: Becoming Leader. State: Replica: 1256088a95854251986461569a79f7d4, State: Running, Role: LEADER
I20260808 15:54:14.022322 29086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256088a95854251986461569a79f7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } }
I20260808 15:54:14.023404 29064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:14.025838 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1256088a95854251986461569a79f7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256088a95854251986461569a79f7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } } }
I20260808 15:54:14.025913 29089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1256088a95854251986461569a79f7d4. Latest consensus state: current_term: 1 leader_uuid: "1256088a95854251986461569a79f7d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1256088a95854251986461569a79f7d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40713 } } }
I20260808 15:54:14.026290 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:14.026374 29089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:14.027366 29094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:14.033594 29094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:14.043715 29094 catalog_manager.cc:1347] Generated new cluster ID: 5cd1b2a531444116a97fe3b299b6a848
I20260808 15:54:14.043861 29094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:14.059679 29094 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:14.060927 29094 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:14.067538 29094 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1256088a95854251986461569a79f7d4: Generated new TSK 0
I20260808 15:54:14.068346 29094 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:14.082180 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28964
W20260808 15:54:14.084338 29055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:55157
W20260808 15:54:14.085177 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 28964: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:14.085366 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 28964
W20260808 15:54:14.116737 28936 connection.cc:535] client connection to 127.0.0.1:40713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40713): Connection reset by peer (error 104)
2026-08-08T15:54:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:14 dist-test-slave-twwt krb5kdc[29120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:14 dist-test-slave-twwt krb5kdc[29120](info): set up 2 sockets
Aug 08 15:54:14 dist-test-slave-twwt krb5kdc[29120](info): commencing operation
krb5kdc: starting...
W20260808 15:54:15.460996 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:15 dist-test-slave-twwt krb5kdc[29120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260808 15:54:15.827013 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:16.190014 29138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:16.190342 29138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:16.190510 29138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:16.197976 29138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:16.198098 29138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:16.198186 29138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:16.198297 29138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:16.198391 29138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:16.209087 29138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:16.210502 29138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:16.211956 29138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:16.224370 29145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:16.224349 29152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:16.224437 29148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:16.225431 29138 server_base.cc:1034] running on GCE node
I20260808 15:54:16.226254 29138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:16.227870 29138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:16.229125 29138 hybrid_clock.cc:648] HybridClock initialized: now 1786204456229093 us; error 59 us; skew 500 ppm
I20260808 15:54:16.229514 29138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:16 dist-test-slave-twwt krb5kdc[29120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:16.233027 29138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260808 15:54:16.234752 29138 webserver.cc:460] Webserver started at http://127.0.0.1:45567/ using document root <none> and password file <none>
I20260808 15:54:16.235491 29138 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:16.235651 29138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:16.235997 29138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:16.239228 29138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8"
format_stamp: "Formatted at 2026-08-08 15:54:16 on dist-test-slave-twwt"
server_key: "6d7804d7fdc183f20cdfb880073f1475"
server_key_iv: "25d61b02e39caef07d7662d413b34793"
server_key_version: "encryptionkey@0"
I20260808 15:54:16.240053 29138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8"
format_stamp: "Formatted at 2026-08-08 15:54:16 on dist-test-slave-twwt"
server_key: "6d7804d7fdc183f20cdfb880073f1475"
server_key_iv: "25d61b02e39caef07d7662d413b34793"
server_key_version: "encryptionkey@0"
I20260808 15:54:16.245700 29138 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:16.249341 29166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:16.250819 29138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260808 15:54:16.251005 29138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8"
format_stamp: "Formatted at 2026-08-08 15:54:16 on dist-test-slave-twwt"
server_key: "6d7804d7fdc183f20cdfb880073f1475"
server_key_iv: "25d61b02e39caef07d7662d413b34793"
server_key_version: "encryptionkey@0"
I20260808 15:54:16.251223 29138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:16.257447 29138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:16.261695 29138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:16.262002 29138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:16.282202 29138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33943
I20260808 15:54:16.282263 29250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33943 every 8 connection(s)
I20260808 15:54:16.283794 29138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:16.288072 29252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:16.293411 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29138
I20260808 15:54:16.293617 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:16.294085 27987 external_mini_cluster.cc:1428] Setting key 47522efdd7eba9d826f592aa2d153e5f
I20260808 15:54:16.301815 29252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8: Bootstrap starting.
I20260808 15:54:16.304147 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:16.296102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:48219 (local address 127.0.0.1:33943)
0808 15:54:16.296994 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:16.297011 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:16.297070 (+ 59us) server_negotiation.cc:407] Connection header received
0808 15:54:16.299880 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:16.299912 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:16.300375 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:16.300866 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:16.301934 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:16.301943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:16.301959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:16.301980 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:16.302064 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:16.302093 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:16.302338 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:16.302497 (+ 159us) server_negotiation.cc:299] Negotiation successful
0808 15:54:16.302743 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":340,"threads_started":1}
I20260808 15:54:16.305872 29252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:16.307308 29252 log.cc:824] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:16.307999 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:16.310256 29252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8: No bootstrap required, opened a new log
I20260808 15:54:16.315392 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:16.312813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.192:55495 (local address 127.0.0.1:33943)
0808 15:54:16.313007 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:16.313014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:16.313336 (+ 322us) server_negotiation.cc:407] Connection header received
0808 15:54:16.313489 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:16.313494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:16.313559 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:16.313666 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:16.314592 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:16.314599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:16.314605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:16.314646 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:16.314694 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:16.314714 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:16.314946 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:16.315003 (+ 57us) server_negotiation.cc:299] Negotiation successful
0808 15:54:16.315069 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260808 15:54:16.319317 29252 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33943 } }
I20260808 15:54:16.319721 29252 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:16.319855 29252 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1e6a2049fe46f094e1fb6d5bc085d8, State: Initialized, Role: FOLLOWER
I20260808 15:54:16.320472 29252 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33943 } }
I20260808 15:54:16.320736 29252 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:16.320868 29252 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:16.321051 29252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:16.324054 29252 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33943 } }
I20260808 15:54:16.324620 29252 leader_election.cc:302] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec1e6a2049fe46f094e1fb6d5bc085d8; no voters:
I20260808 15:54:16.325232 29252 leader_election.cc:288] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:16.325433 29269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:16.326270 29269 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [term 1 LEADER]: Becoming Leader. State: Replica: ec1e6a2049fe46f094e1fb6d5bc085d8, State: Running, Role: LEADER
I20260808 15:54:16.326923 29269 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33943 } }
I20260808 15:54:16.327865 29252 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:16.330515 29271 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec1e6a2049fe46f094e1fb6d5bc085d8. Latest consensus state: current_term: 1 leader_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33943 } } }
I20260808 15:54:16.331051 29271 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:16.330506 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec1e6a2049fe46f094e1fb6d5bc085d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33943 } } }
I20260808 15:54:16.331712 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:16.339370 29277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:16.344425 29277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:16.353448 29277 catalog_manager.cc:1347] Generated new cluster ID: d584a0b2555f4d18b9479bb5fe3a57d6
I20260808 15:54:16.353588 29277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:16.383749 29277 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:16.384282 29277 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:16.394068 29277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec1e6a2049fe46f094e1fb6d5bc085d8: Generated new TSK 0
I20260808 15:54:16.394977 29277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:16.407106 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29138
W20260808 15:54:16.408223 29242 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.192:48219
W20260808 15:54:16.408960 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29138: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:16.409116 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29138
2026-08-08T15:54:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:16 dist-test-slave-twwt krb5kdc[29297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:16 dist-test-slave-twwt krb5kdc[29297](info): set up 2 sockets
Aug 08 15:54:16 dist-test-slave-twwt krb5kdc[29297](info): commencing operation
krb5kdc: starting...
W20260808 15:54:17.764323 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.007s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:17 dist-test-slave-twwt krb5kdc[29297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:18.138547 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:35719
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:35719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:18.496860 29320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:18.497161 29320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:18.497363 29320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:18.504602 29320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:18.504707 29320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:18.504753 29320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:18.504848 29320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:18.505029 29320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:18.516366 29320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:35719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:35719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:18.517911 29320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:18.519497 29320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:18.533010 29331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:18.533025 29334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:18.533771 29330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:18.534252 29320 server_base.cc:1034] running on GCE node
I20260808 15:54:18.535115 29320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:18.536561 29320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:18.537798 29320 hybrid_clock.cc:648] HybridClock initialized: now 1786204458537754 us; error 43 us; skew 500 ppm
I20260808 15:54:18.538163 29320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:18 dist-test-slave-twwt krb5kdc[29297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204458, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:18.541543 29320 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:18.543175 29320 webserver.cc:460] Webserver started at http://127.27.84.254:41597/ using document root <none> and password file <none>
I20260808 15:54:18.543902 29320 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:18.544070 29320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:18.544389 29320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:18.547540 29320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b"
format_stamp: "Formatted at 2026-08-08 15:54:18 on dist-test-slave-twwt"
server_key: "824dc56102eda87993b47eab1df60a7c"
server_key_iv: "9e32afd843305dea80dee6827d444677"
server_key_version: "encryptionkey@0"
I20260808 15:54:18.548337 29320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b"
format_stamp: "Formatted at 2026-08-08 15:54:18 on dist-test-slave-twwt"
server_key: "824dc56102eda87993b47eab1df60a7c"
server_key_iv: "9e32afd843305dea80dee6827d444677"
server_key_version: "encryptionkey@0"
I20260808 15:54:18.553661 29320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:18.557255 29344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:18.558961 29320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260808 15:54:18.559185 29320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b"
format_stamp: "Formatted at 2026-08-08 15:54:18 on dist-test-slave-twwt"
server_key: "824dc56102eda87993b47eab1df60a7c"
server_key_iv: "9e32afd843305dea80dee6827d444677"
server_key_version: "encryptionkey@0"
I20260808 15:54:18.559425 29320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:18.580433 29320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:18.585253 29320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:18.585613 29320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:18.605209 29320 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:35719
I20260808 15:54:18.605175 29424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:35719 every 8 connection(s)
I20260808 15:54:18.606797 29320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:18.610831 29426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:18.614678 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29320
I20260808 15:54:18.614903 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:18.615353 27987 external_mini_cluster.cc:1428] Setting key a867ef4b28c78253b99e548137dc2056
I20260808 15:54:18.624842 29426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b: Bootstrap starting.
I20260808 15:54:18.628610 29426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:18.629670 29426 log.cc:824] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:54:18 dist-test-slave-twwt krb5kdc[29297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:18.633009 29426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b: No bootstrap required, opened a new log
I20260808 15:54:18.640424 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:18.617384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38033 (local address 127.27.84.254:35719)
0808 15:54:18.618239 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:18.618256 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:18.618310 (+ 54us) server_negotiation.cc:407] Connection header received
0808 15:54:18.620320 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:18.620354 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:18.620944 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:18.621415 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:18.622841 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:18.624864 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:18.626225 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:18.627367 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:18.627549 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:18.632236 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:18.632244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:18.632258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:18.632298 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:18.635644 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:18.636740 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:18.636746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:18.636752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:18.636958 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:18.637474 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:18.637477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:18.637479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:18.638056 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:18.638297 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:18.638968 (+ 671us) server_negotiation.cc:299] Negotiation successful
0808 15:54:18.639356 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":376,"threads_started":1}
I20260808 15:54:18.642166 29426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 35719 } }
I20260808 15:54:18.642558 29426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:18.642704 29426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bd0a97a2394dfc90e42cb6e4e7d25b, State: Initialized, Role: FOLLOWER
I20260808 15:54:18.643397 29426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 35719 } }
I20260808 15:54:18.643678 29426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:18.643836 29426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:18.644028 29426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:18.644498 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:18.647269 29426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 35719 } }
I20260808 15:54:18.647933 29426 leader_election.cc:302] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78bd0a97a2394dfc90e42cb6e4e7d25b; no voters:
I20260808 15:54:18.648689 29426 leader_election.cc:288] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:18.648814 29438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:18.650552 29438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [term 1 LEADER]: Becoming Leader. State: Replica: 78bd0a97a2394dfc90e42cb6e4e7d25b, State: Running, Role: LEADER
I20260808 15:54:18.651260 29438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 35719 } }
I20260808 15:54:18.651834 29426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:18.654577 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78bd0a97a2394dfc90e42cb6e4e7d25b. Latest consensus state: current_term: 1 leader_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 35719 } } }
I20260808 15:54:18.655411 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:18.655409 29441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78bd0a97a2394dfc90e42cb6e4e7d25b" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 35719 } } }
I20260808 15:54:18.659163 29441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:18.662019 29451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:18.668766 29451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:18.671892 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:18.650194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51485 (local address 127.27.84.254:35719)
0808 15:54:18.650417 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:18.650426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:18.651480 (+ 1054us) server_negotiation.cc:407] Connection header received
0808 15:54:18.651659 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:18.651665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:18.651742 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:18.651869 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:18.653988 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:18.655882 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:18.658177 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:18.659246 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:18.659418 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:18.663683 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:18.663696 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:18.663707 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:18.663754 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:18.668017 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:18.669051 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:18.669057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:18.669060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:18.669217 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:18.669873 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:18.669879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:18.669882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:18.670305 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:18.670564 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:18.671306 (+ 742us) server_negotiation.cc:299] Negotiation successful
0808 15:54:18.671507 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:54:18.682879 29451 catalog_manager.cc:1347] Generated new cluster ID: 0538fc049007451cabc0c26c31195bdb
I20260808 15:54:18.683009 29451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:18.702561 29451 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:18.704447 29451 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:18.713742 29451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78bd0a97a2394dfc90e42cb6e4e7d25b: Generated new TSK 0
I20260808 15:54:18.714880 29451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:18.729763 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29320
I20260808 15:54:18.731225 29414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:18.824695 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29320
W20260808 15:54:18.852842 29295 connection.cc:535] client connection to 127.27.84.254:35719 recv error: Network error: failed to read from TLS socket (remote: 127.27.84.254:35719): Connection reset by peer (error 104)
2026-08-08T15:54:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:18 dist-test-slave-twwt krb5kdc[29473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:18 dist-test-slave-twwt krb5kdc[29473](info): set up 2 sockets
Aug 08 15:54:18 dist-test-slave-twwt krb5kdc[29473](info): commencing operation
krb5kdc: starting...
W20260808 15:54:20.190497 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:20 dist-test-slave-twwt krb5kdc[29473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:20.548233 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36231
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:36231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:20.911281 29490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:20.911582 29490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:20.911810 29490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:20.919166 29490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:20.919281 29490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:20.919401 29490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:20.919514 29490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:20.919598 29490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:20.929540 29490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:36231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:36231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:20.930966 29490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:20.932382 29490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:20.944749 29499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:20.945050 29498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:20.945075 29501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:20.945770 29490 server_base.cc:1034] running on GCE node
I20260808 15:54:20.946583 29490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:20.948092 29490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:20.949303 29490 hybrid_clock.cc:648] HybridClock initialized: now 1786204460949265 us; error 41 us; skew 500 ppm
I20260808 15:54:20.949678 29490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:20 dist-test-slave-twwt krb5kdc[29473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204460, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:20.958475 29490 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:20.960127 29490 webserver.cc:460] Webserver started at http://127.27.84.254:40243/ using document root <none> and password file <none>
I20260808 15:54:20.960867 29490 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:20.961030 29490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:20.961354 29490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:20.964521 29490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "aa2f814b09284936b79f0ea53de607b3"
format_stamp: "Formatted at 2026-08-08 15:54:20 on dist-test-slave-twwt"
server_key: "aa65c5bf38d38ba1fd4570083deac0b2"
server_key_iv: "56e7da87926e9a8fdcd2d3c576a9b050"
server_key_version: "encryptionkey@0"
I20260808 15:54:20.965323 29490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "aa2f814b09284936b79f0ea53de607b3"
format_stamp: "Formatted at 2026-08-08 15:54:20 on dist-test-slave-twwt"
server_key: "aa65c5bf38d38ba1fd4570083deac0b2"
server_key_iv: "56e7da87926e9a8fdcd2d3c576a9b050"
server_key_version: "encryptionkey@0"
I20260808 15:54:20.970834 29490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:54:20.974406 29514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:20.975875 29490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:20.976074 29490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "aa2f814b09284936b79f0ea53de607b3"
format_stamp: "Formatted at 2026-08-08 15:54:20 on dist-test-slave-twwt"
server_key: "aa65c5bf38d38ba1fd4570083deac0b2"
server_key_iv: "56e7da87926e9a8fdcd2d3c576a9b050"
server_key_version: "encryptionkey@0"
I20260808 15:54:20.976287 29490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:20.999826 29490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:21.004658 29490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:21.004993 29490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:21.024993 29490 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:36231
I20260808 15:54:21.025012 29585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:36231 every 8 connection(s)
I20260808 15:54:21.026479 29490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:21.030391 29586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:21.033622 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29490
I20260808 15:54:21.033819 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:21.034369 27987 external_mini_cluster.cc:1428] Setting key 804fef9512f9a18bd76f5a2217c0ea98
I20260808 15:54:21.043493 29586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3: Bootstrap starting.
I20260808 15:54:21.047432 29586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:21.048584 29586 log.cc:824] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:21.049016 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:21.036537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48207 (local address 127.27.84.254:36231)
0808 15:54:21.037506 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:21.037519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:21.037563 (+ 44us) server_negotiation.cc:407] Connection header received
0808 15:54:21.039392 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:21.039417 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:21.039876 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:21.040233 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:21.040927 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:21.042249 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:21.044178 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:21.044842 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:21.045026 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:21.046439 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:21.046447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:21.046461 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:21.047093 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:21.047169 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:21.047194 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:21.047322 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:21.047718 (+ 396us) server_negotiation.cc:299] Negotiation successful
0808 15:54:21.047956 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":461,"threads_started":1}
I20260808 15:54:21.051707 29586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3: No bootstrap required, opened a new log
I20260808 15:54:21.053171 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:21.062129 29586 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2f814b09284936b79f0ea53de607b3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36231 } }
I20260808 15:54:21.062558 29586 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:21.062693 29586 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa2f814b09284936b79f0ea53de607b3, State: Initialized, Role: FOLLOWER
I20260808 15:54:21.063350 29586 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2f814b09284936b79f0ea53de607b3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36231 } }
I20260808 15:54:21.063616 29586 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:21.063786 29586 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:21.063966 29586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:21.065532 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:21.058413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58105 (local address 127.27.84.254:36231)
0808 15:54:21.058645 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:21.058652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:21.059211 (+ 559us) server_negotiation.cc:407] Connection header received
0808 15:54:21.059347 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:21.059353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:21.059435 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:21.059555 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:21.060048 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:21.061236 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:21.062699 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:21.063305 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:21.063457 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:21.064720 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:21.064730 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:21.064737 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:21.064755 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:21.064817 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:21.064836 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:21.065013 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:21.065180 (+ 167us) server_negotiation.cc:299] Negotiation successful
0808 15:54:21.065251 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:54:21.066710 29586 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2f814b09284936b79f0ea53de607b3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36231 } }
I20260808 15:54:21.067373 29586 leader_election.cc:302] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa2f814b09284936b79f0ea53de607b3; no voters:
I20260808 15:54:21.067922 29586 leader_election.cc:288] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:21.068089 29607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:21.068848 29607 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [term 1 LEADER]: Becoming Leader. State: Replica: aa2f814b09284936b79f0ea53de607b3, State: Running, Role: LEADER
I20260808 15:54:21.069506 29607 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2f814b09284936b79f0ea53de607b3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36231 } }
I20260808 15:54:21.070206 29586 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:21.072570 29610 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa2f814b09284936b79f0ea53de607b3. Latest consensus state: current_term: 1 leader_uuid: "aa2f814b09284936b79f0ea53de607b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2f814b09284936b79f0ea53de607b3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36231 } } }
I20260808 15:54:21.073071 29610 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:21.073980 29608 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa2f814b09284936b79f0ea53de607b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa2f814b09284936b79f0ea53de607b3" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 36231 } } }
I20260808 15:54:21.074426 29608 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:21.076267 29615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:21.082021 29615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:21.090543 29615 catalog_manager.cc:1347] Generated new cluster ID: 65f51d9553ae46ac8e02365d5186dd80
I20260808 15:54:21.090692 29615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:21.113405 29615 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:21.114817 29615 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:21.123107 29615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa2f814b09284936b79f0ea53de607b3: Generated new TSK 0
I20260808 15:54:21.123997 29615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:21.143221 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29490
W20260808 15:54:21.144685 29579 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48207
W20260808 15:54:21.145570 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29490: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:21.145761 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29490
W20260808 15:54:21.172752 29471 connection.cc:535] client connection to 127.27.84.254:36231 recv error: Network error: failed to read from TLS socket (remote: 127.27.84.254:36231): Connection reset by peer (error 104)
2026-08-08T15:54:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:21 dist-test-slave-twwt krb5kdc[29642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:21 dist-test-slave-twwt krb5kdc[29642](info): set up 2 sockets
Aug 08 15:54:21 dist-test-slave-twwt krb5kdc[29642](info): commencing operation
krb5kdc: starting...
W20260808 15:54:22.520845 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.004s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:22 dist-test-slave-twwt krb5kdc[29642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:22.894238 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43991
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:45163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:43991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:23.268399 29659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:23.268694 29659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:23.268880 29659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:23.276289 29659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:23.276398 29659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:23.276512 29659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:23.276597 29659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:23.276664 29659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:23.286648 29659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:45163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:43991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:43991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:23.288012 29659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:23.289476 29659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:23.301919 29669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:23.302124 29672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:23.301987 29670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:23.302862 29659 server_base.cc:1034] running on GCE node
I20260808 15:54:23.303757 29659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:23.305181 29659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:23.306421 29659 hybrid_clock.cc:648] HybridClock initialized: now 1786204463306364 us; error 51 us; skew 500 ppm
I20260808 15:54:23.306835 29659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:23 dist-test-slave-twwt krb5kdc[29642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204463, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:23.310076 29659 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:23.311689 29659 webserver.cc:460] Webserver started at http://127.27.84.254:37219/ using document root <none> and password file <none>
I20260808 15:54:23.312353 29659 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:23.312459 29659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:23.312733 29659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:23.315824 29659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "3e9c9bf78f0642b5a48549766b0e165d"
format_stamp: "Formatted at 2026-08-08 15:54:23 on dist-test-slave-twwt"
server_key: "9c28228ad6d78ae73883962bf0e7d109"
server_key_iv: "0548829937d325177278e5dbb21ff2af"
server_key_version: "encryptionkey@0"
I20260808 15:54:23.316556 29659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "3e9c9bf78f0642b5a48549766b0e165d"
format_stamp: "Formatted at 2026-08-08 15:54:23 on dist-test-slave-twwt"
server_key: "9c28228ad6d78ae73883962bf0e7d109"
server_key_iv: "0548829937d325177278e5dbb21ff2af"
server_key_version: "encryptionkey@0"
I20260808 15:54:23.321907 29659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:54:23.325394 29686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:23.326862 29659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:23.327049 29659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "3e9c9bf78f0642b5a48549766b0e165d"
format_stamp: "Formatted at 2026-08-08 15:54:23 on dist-test-slave-twwt"
server_key: "9c28228ad6d78ae73883962bf0e7d109"
server_key_iv: "0548829937d325177278e5dbb21ff2af"
server_key_version: "encryptionkey@0"
I20260808 15:54:23.327261 29659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:23.333305 29659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:23.337520 29659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:23.337836 29659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:23.356196 29659 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:43991
I20260808 15:54:23.356233 29761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:43991 every 8 connection(s)
I20260808 15:54:23.357625 29659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:23.361500 29762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:23.362908 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29659
I20260808 15:54:23.363104 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:23.363575 27987 external_mini_cluster.cc:1428] Setting key b60208a0fcfda0cd12a9bc01dacdfb23
I20260808 15:54:23.371655 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:23.365563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53985 (local address 127.27.84.254:43991)
0808 15:54:23.366448 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:23.366460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:23.366508 (+ 48us) server_negotiation.cc:407] Connection header received
0808 15:54:23.368166 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:23.368191 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:23.368542 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:23.368900 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:23.370061 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:23.370068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:23.370081 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:23.370096 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:23.370157 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:23.370180 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:23.370346 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:23.370495 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 15:54:23.370736 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":430,"threads_started":1}
I20260808 15:54:23.375922 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:23.376240 29762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d: Bootstrap starting.
I20260808 15:54:23.379400 29762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:23.380568 29762 log.cc:824] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:23.383214 29768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:23.380458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37227 (local address 127.27.84.254:43991)
0808 15:54:23.380674 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:23.380683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:23.381202 (+ 519us) server_negotiation.cc:407] Connection header received
0808 15:54:23.381418 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:23.381428 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:23.381490 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:23.381615 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:23.382509 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:23.382513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:23.382516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:23.382530 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:23.382568 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:23.382582 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:23.382725 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:23.382869 (+ 144us) server_negotiation.cc:299] Negotiation successful
0808 15:54:23.382915 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260808 15:54:23.384162 29762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d: No bootstrap required, opened a new log
I20260808 15:54:23.391891 29762 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9c9bf78f0642b5a48549766b0e165d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43991 } }
I20260808 15:54:23.392299 29762 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:23.392444 29762 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9c9bf78f0642b5a48549766b0e165d, State: Initialized, Role: FOLLOWER
I20260808 15:54:23.393038 29762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9c9bf78f0642b5a48549766b0e165d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43991 } }
I20260808 15:54:23.393301 29762 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:23.393467 29762 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:23.393694 29762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:23.396147 29762 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9c9bf78f0642b5a48549766b0e165d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43991 } }
I20260808 15:54:23.396667 29762 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9c9bf78f0642b5a48549766b0e165d; no voters:
I20260808 15:54:23.397321 29762 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:23.397498 29778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:23.398234 29778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [term 1 LEADER]: Becoming Leader. State: Replica: 3e9c9bf78f0642b5a48549766b0e165d, State: Running, Role: LEADER
I20260808 15:54:23.398859 29778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9c9bf78f0642b5a48549766b0e165d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43991 } }
I20260808 15:54:23.399653 29762 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:23.402024 29781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9c9bf78f0642b5a48549766b0e165d. Latest consensus state: current_term: 1 leader_uuid: "3e9c9bf78f0642b5a48549766b0e165d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9c9bf78f0642b5a48549766b0e165d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43991 } } }
I20260808 15:54:23.402132 29780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9c9bf78f0642b5a48549766b0e165d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9c9bf78f0642b5a48549766b0e165d" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 43991 } } }
I20260808 15:54:23.402491 29781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:23.402532 29780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:23.404726 29784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:23.410344 29784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:23.420187 29784 catalog_manager.cc:1347] Generated new cluster ID: 5d647bc6b1fb49128757f89167a596c8
I20260808 15:54:23.420333 29784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:23.438408 29784 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:23.438982 29784 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:23.465054 29784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9c9bf78f0642b5a48549766b0e165d: Generated new TSK 0
I20260808 15:54:23.466229 29784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:23.474654 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29659
W20260808 15:54:23.476037 29757 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53985
W20260808 15:54:23.476807 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:23.476969 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29659
2026-08-08T15:54:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:23 dist-test-slave-twwt krb5kdc[29807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:23 dist-test-slave-twwt krb5kdc[29807](info): set up 2 sockets
Aug 08 15:54:23 dist-test-slave-twwt krb5kdc[29807](info): commencing operation
krb5kdc: starting...
W20260808 15:54:24.844717 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.322s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:25 dist-test-slave-twwt krb5kdc[29807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:25.218545 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34785
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:33257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:34785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:25.596153 29826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:25.596460 29826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:25.596623 29826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:25.604152 29826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:25.604266 29826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:25.604377 29826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:25.604487 29826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:25.604575 29826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:25.614979 29826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:33257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:34785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:34785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:25.616351 29826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:25.617745 29826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:25.629976 29834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:25.630187 29836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:25.630540 29839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:25.631217 29826 server_base.cc:1034] running on GCE node
I20260808 15:54:25.632097 29826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:25.633622 29826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:25.634877 29826 hybrid_clock.cc:648] HybridClock initialized: now 1786204465634811 us; error 57 us; skew 500 ppm
I20260808 15:54:25.635279 29826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:25 dist-test-slave-twwt krb5kdc[29807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204465, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:25.638813 29826 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:25.640413 29826 webserver.cc:460] Webserver started at http://127.27.84.254:38107/ using document root <none> and password file <none>
I20260808 15:54:25.641168 29826 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:25.641319 29826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:25.641638 29826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:25.644954 29826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "58cc594689274681b0e1c0679d97ca53"
format_stamp: "Formatted at 2026-08-08 15:54:25 on dist-test-slave-twwt"
server_key: "19c6ab9b6c21ad8014beac7013d838f0"
server_key_iv: "4ace9cae65547476d748dddc054754ef"
server_key_version: "encryptionkey@0"
I20260808 15:54:25.645826 29826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "58cc594689274681b0e1c0679d97ca53"
format_stamp: "Formatted at 2026-08-08 15:54:25 on dist-test-slave-twwt"
server_key: "19c6ab9b6c21ad8014beac7013d838f0"
server_key_iv: "4ace9cae65547476d748dddc054754ef"
server_key_version: "encryptionkey@0"
I20260808 15:54:25.651754 29826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260808 15:54:25.655783 29847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:25.657198 29826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:54:25.657419 29826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "58cc594689274681b0e1c0679d97ca53"
format_stamp: "Formatted at 2026-08-08 15:54:25 on dist-test-slave-twwt"
server_key: "19c6ab9b6c21ad8014beac7013d838f0"
server_key_iv: "4ace9cae65547476d748dddc054754ef"
server_key_version: "encryptionkey@0"
I20260808 15:54:25.657647 29826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:25.692255 29826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:25.696772 29826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:25.697099 29826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:25.717051 29826 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:34785
I20260808 15:54:25.717115 29934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:34785 every 8 connection(s)
I20260808 15:54:25.718580 29826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:25.722573 29935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:25.725771 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 29826
I20260808 15:54:25.725973 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:25.726526 27987 external_mini_cluster.cc:1428] Setting key 33ec81b1460b87aa3e94865a39f212da
I20260808 15:54:25.735947 29935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53: Bootstrap starting.
I20260808 15:54:25.740337 29935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:25.742128 29935 log.cc:824] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:54:25 dist-test-slave-twwt krb5kdc[29807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:25.745851 29935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53: No bootstrap required, opened a new log
I20260808 15:54:25.752655 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:25.728585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41987 (local address 127.27.84.254:34785)
0808 15:54:25.729457 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:25.729469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:25.729511 (+ 42us) server_negotiation.cc:407] Connection header received
0808 15:54:25.731346 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:25.731371 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:25.731802 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:25.732175 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:25.734136 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:25.735585 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:25.737494 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:25.738319 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:25.738473 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:25.744169 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:25.744179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:25.744197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:25.744265 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:25.747716 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:25.748736 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:25.748744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:25.748751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:25.748979 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:25.749491 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:25.749500 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:25.749503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:25.750181 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:25.750425 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:25.750983 (+ 558us) server_negotiation.cc:299] Negotiation successful
0808 15:54:25.751448 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":420,"threads_started":1}
I20260808 15:54:25.755812 29935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc594689274681b0e1c0679d97ca53" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34785 } }
I20260808 15:54:25.756237 29935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:25.756372 29935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58cc594689274681b0e1c0679d97ca53, State: Initialized, Role: FOLLOWER
I20260808 15:54:25.757052 29935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc594689274681b0e1c0679d97ca53" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34785 } }
I20260808 15:54:25.757294 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:25.757367 29935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:25.757578 29935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:25.757764 29935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:25.760175 29935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc594689274681b0e1c0679d97ca53" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34785 } }
I20260808 15:54:25.760903 29935 leader_election.cc:302] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58cc594689274681b0e1c0679d97ca53; no voters:
I20260808 15:54:25.761703 29935 leader_election.cc:288] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:25.761809 29949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:25.763485 29949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [term 1 LEADER]: Becoming Leader. State: Replica: 58cc594689274681b0e1c0679d97ca53, State: Running, Role: LEADER
I20260808 15:54:25.764132 29949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc594689274681b0e1c0679d97ca53" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34785 } }
I20260808 15:54:25.764539 29935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:25.767445 29951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58cc594689274681b0e1c0679d97ca53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc594689274681b0e1c0679d97ca53" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34785 } } }
I20260808 15:54:25.768136 29951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:25.768721 29950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58cc594689274681b0e1c0679d97ca53. Latest consensus state: current_term: 1 leader_uuid: "58cc594689274681b0e1c0679d97ca53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58cc594689274681b0e1c0679d97ca53" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 34785 } } }
I20260808 15:54:25.769109 29950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:25.780920 29959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:25.783363 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:25.762272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57635 (local address 127.27.84.254:34785)
0808 15:54:25.762557 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:25.762563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:25.763144 (+ 581us) server_negotiation.cc:407] Connection header received
0808 15:54:25.763476 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:25.763482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:25.763567 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:25.763706 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:25.765565 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:25.766778 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:25.768753 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:25.769521 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:25.769691 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:25.774331 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:25.774342 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:25.774351 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:25.774417 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:25.779369 (+ 4952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:25.780522 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:25.780528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:25.780532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:25.780712 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:25.781245 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:25.781251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:25.781254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:25.781636 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:25.781924 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:25.782917 (+ 993us) server_negotiation.cc:299] Negotiation successful
0808 15:54:25.783108 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260808 15:54:25.789909 29959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:25.800365 29959 catalog_manager.cc:1347] Generated new cluster ID: 27326783970940ed8214ef99dabc8901
I20260808 15:54:25.800508 29959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:25.823300 29959 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:25.825577 29959 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:25.833932 29959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58cc594689274681b0e1c0679d97ca53: Generated new TSK 0
I20260808 15:54:25.834893 29959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:25.847180 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 29826
I20260808 15:54:25.848649 29928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:25.942216 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 29826
2026-08-08T15:54:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:26 dist-test-slave-twwt krb5kdc[29987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:26 dist-test-slave-twwt krb5kdc[29987](info): set up 2 sockets
Aug 08 15:54:26 dist-test-slave-twwt krb5kdc[29987](info): commencing operation
krb5kdc: starting...
W20260808 15:54:27.305166 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:27 dist-test-slave-twwt krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:27.671300 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:39273
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:43135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:39273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:28.036890 30013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:28.037278 30013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:28.037442 30013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:28.045117 30013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:28.045291 30013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:28.045397 30013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:28.045511 30013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:28.045585 30013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:28.055894 30013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:43135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:39273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:28.057268 30013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:28.058660 30013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:28.071242 30027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:28.071296 30023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:28.071190 30022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:28.072209 30013 server_base.cc:1034] running on GCE node
I20260808 15:54:28.073069 30013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:28.074499 30013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:28.075742 30013 hybrid_clock.cc:648] HybridClock initialized: now 1786204468075686 us; error 47 us; skew 500 ppm
I20260808 15:54:28.076170 30013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:28 dist-test-slave-twwt krb5kdc[29987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:28.079640 30013 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:28.081256 30013 webserver.cc:460] Webserver started at http://127.27.84.254:44899/ using document root <none> and password file <none>
I20260808 15:54:28.081990 30013 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:28.082145 30013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:28.082460 30013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:28.085754 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "1e90308be7484dc0b9e9e76030ffd068"
format_stamp: "Formatted at 2026-08-08 15:54:28 on dist-test-slave-twwt"
server_key: "26d3e69003ceda027ac3b4af9ed7b18b"
server_key_iv: "ff08e8b6eb2fd664f4a45573b88d3bf1"
server_key_version: "encryptionkey@0"
I20260808 15:54:28.086771 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "1e90308be7484dc0b9e9e76030ffd068"
format_stamp: "Formatted at 2026-08-08 15:54:28 on dist-test-slave-twwt"
server_key: "26d3e69003ceda027ac3b4af9ed7b18b"
server_key_iv: "ff08e8b6eb2fd664f4a45573b88d3bf1"
server_key_version: "encryptionkey@0"
I20260808 15:54:28.092087 30013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260808 15:54:28.095824 30038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:28.097314 30013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260808 15:54:28.097556 30013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "1e90308be7484dc0b9e9e76030ffd068"
format_stamp: "Formatted at 2026-08-08 15:54:28 on dist-test-slave-twwt"
server_key: "26d3e69003ceda027ac3b4af9ed7b18b"
server_key_iv: "ff08e8b6eb2fd664f4a45573b88d3bf1"
server_key_version: "encryptionkey@0"
I20260808 15:54:28.097843 30013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:28.112450 30013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:28.116983 30013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:28.117241 30013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:28.136790 30013 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:39273
I20260808 15:54:28.136821 30125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:39273 every 8 connection(s)
I20260808 15:54:28.138296 30013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:28.142359 30126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:28.146466 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30013
I20260808 15:54:28.146670 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:28.147131 27987 external_mini_cluster.cc:1428] Setting key 0cf9ccba29e4f02850e99e85b4fd9ba1
I20260808 15:54:28.155710 30126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068: Bootstrap starting.
I20260808 15:54:28.159746 30126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:28.160873 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:28.148927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45893 (local address 127.27.84.254:39273)
0808 15:54:28.149733 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:28.149745 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:28.149790 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:54:28.151524 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:28.151549 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:28.151991 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:28.152380 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:28.153068 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:28.154387 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:28.156296 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:28.156986 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:28.157146 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:28.158474 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:28.158481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:28.158496 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:28.159121 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:28.159194 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:28.159220 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:28.159310 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:28.159563 (+ 253us) server_negotiation.cc:299] Negotiation successful
0808 15:54:28.159851 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":371,"threads_started":1}
I20260808 15:54:28.160965 30126 log.cc:824] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:28.164323 30126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068: No bootstrap required, opened a new log
I20260808 15:54:28.164917 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:28.173832 30126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e90308be7484dc0b9e9e76030ffd068" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39273 } }
I20260808 15:54:28.174301 30126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:28.174438 30126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e90308be7484dc0b9e9e76030ffd068, State: Initialized, Role: FOLLOWER
I20260808 15:54:28.175200 30126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e90308be7484dc0b9e9e76030ffd068" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39273 } }
I20260808 15:54:28.175525 30126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:28.175685 30126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:28.175896 30126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:28.176397 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:28.169446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50875 (local address 127.27.84.254:39273)
0808 15:54:28.169684 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:28.169691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:28.170042 (+ 351us) server_negotiation.cc:407] Connection header received
0808 15:54:28.170240 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:28.170244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:28.170312 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:28.170408 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:28.171103 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:28.171845 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:28.173775 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:28.174197 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:28.174338 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:28.175563 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:28.175568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:28.175571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:28.175608 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:28.175656 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:28.175684 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:28.175764 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:28.175937 (+ 173us) server_negotiation.cc:299] Negotiation successful
0808 15:54:28.176038 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260808 15:54:28.178470 30126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e90308be7484dc0b9e9e76030ffd068" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39273 } }
I20260808 15:54:28.179039 30126 leader_election.cc:302] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e90308be7484dc0b9e9e76030ffd068; no voters:
I20260808 15:54:28.179620 30126 leader_election.cc:288] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:28.179769 30144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:28.181396 30144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [term 1 LEADER]: Becoming Leader. State: Replica: 1e90308be7484dc0b9e9e76030ffd068, State: Running, Role: LEADER
I20260808 15:54:28.181993 30126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:28.181996 30144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e90308be7484dc0b9e9e76030ffd068" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39273 } }
I20260808 15:54:28.185161 30146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e90308be7484dc0b9e9e76030ffd068. Latest consensus state: current_term: 1 leader_uuid: "1e90308be7484dc0b9e9e76030ffd068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e90308be7484dc0b9e9e76030ffd068" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39273 } } }
I20260808 15:54:28.185566 30146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:28.185271 30145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e90308be7484dc0b9e9e76030ffd068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e90308be7484dc0b9e9e76030ffd068" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 39273 } } }
I20260808 15:54:28.185696 30145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:28.196465 30153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:28.200892 30153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:28.210237 30153 catalog_manager.cc:1347] Generated new cluster ID: c151f4093fe744d69d397fbd6294ade0
I20260808 15:54:28.210376 30153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:28.227393 30153 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:28.229432 30153 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:28.237686 30153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e90308be7484dc0b9e9e76030ffd068: Generated new TSK 0
I20260808 15:54:28.238921 30153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:28.246500 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30013
W20260808 15:54:28.247745 30121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45893
W20260808 15:54:28.248481 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30013: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:28.248642 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30013
2026-08-08T15:54:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:28 dist-test-slave-twwt krb5kdc[30171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:28 dist-test-slave-twwt krb5kdc[30171](info): set up 2 sockets
Aug 08 15:54:28 dist-test-slave-twwt krb5kdc[30171](info): commencing operation
krb5kdc: starting...
W20260808 15:54:29.610538 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.313s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:29 dist-test-slave-twwt krb5kdc[30171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:29.982995 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:40513
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:46419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:40513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:30.342372 30193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:30.342754 30193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:30.342931 30193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:30.350248 30193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:30.350360 30193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:30.350482 30193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:30.350572 30193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:30.350656 30193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:30.361016 30193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:46419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:40513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:40513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:30.362345 30193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:30.363754 30193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:30.376242 30204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:30.376209 30206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:30.376189 30208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:30.377332 30193 server_base.cc:1034] running on GCE node
I20260808 15:54:30.378294 30193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:30.379897 30193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:30.381179 30193 hybrid_clock.cc:648] HybridClock initialized: now 1786204470381115 us; error 62 us; skew 500 ppm
I20260808 15:54:30.381564 30193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:30 dist-test-slave-twwt krb5kdc[30171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:30.384881 30193 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:30.386435 30193 webserver.cc:460] Webserver started at http://127.27.84.254:34375/ using document root <none> and password file <none>
I20260808 15:54:30.387197 30193 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:30.387357 30193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:30.387673 30193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:30.390901 30193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "dac4e9ddb8a64f96b294152eead317c1"
format_stamp: "Formatted at 2026-08-08 15:54:30 on dist-test-slave-twwt"
server_key: "3d7790cd1513fed10a60dfde1202144c"
server_key_iv: "4967de41101c5ee445e5ed4874ab6868"
server_key_version: "encryptionkey@0"
I20260808 15:54:30.391789 30193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "dac4e9ddb8a64f96b294152eead317c1"
format_stamp: "Formatted at 2026-08-08 15:54:30 on dist-test-slave-twwt"
server_key: "3d7790cd1513fed10a60dfde1202144c"
server_key_iv: "4967de41101c5ee445e5ed4874ab6868"
server_key_version: "encryptionkey@0"
I20260808 15:54:30.397131 30193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260808 15:54:30.400687 30217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:30.402000 30193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:54:30.402189 30193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "dac4e9ddb8a64f96b294152eead317c1"
format_stamp: "Formatted at 2026-08-08 15:54:30 on dist-test-slave-twwt"
server_key: "3d7790cd1513fed10a60dfde1202144c"
server_key_iv: "4967de41101c5ee445e5ed4874ab6868"
server_key_version: "encryptionkey@0"
I20260808 15:54:30.402400 30193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:30.408023 30193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:30.412027 30193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:30.412341 30193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:30.431784 30193 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:40513
I20260808 15:54:30.431818 30290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:40513 every 8 connection(s)
I20260808 15:54:30.433342 30193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:30.437327 30291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:30.439795 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30193
I20260808 15:54:30.440001 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:30.440554 27987 external_mini_cluster.cc:1428] Setting key 175dbae73f39d4fb204af5f438283e66
I20260808 15:54:30.448433 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:30.442401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50637 (local address 127.27.84.254:40513)
0808 15:54:30.443284 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:30.443296 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:30.443341 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:54:30.445008 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:30.445030 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:30.445375 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:30.445698 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:30.446799 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:30.446805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:30.446818 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:30.446832 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:30.446897 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:30.446920 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:30.447096 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:30.447230 (+ 134us) server_negotiation.cc:299] Negotiation successful
0808 15:54:30.447466 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":436,"threads_started":1}
I20260808 15:54:30.449968 30291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1: Bootstrap starting.
I20260808 15:54:30.452150 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:30.453552 30291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:30.454885 30291 log.cc:824] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:30.458631 30291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1: No bootstrap required, opened a new log
I20260808 15:54:30.460279 30295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:30.457680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51319 (local address 127.27.84.254:40513)
0808 15:54:30.457877 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:30.457883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:30.458302 (+ 419us) server_negotiation.cc:407] Connection header received
0808 15:54:30.458450 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:30.458455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:30.458512 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:30.458669 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:30.459635 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:30.459640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:30.459645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0808 15:54:30.459662 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:30.459708 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0808 15:54:30.459727 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:30.459858 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:30.459938 (+ 80us) server_negotiation.cc:299] Negotiation successful
0808 15:54:30.459997 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:54:30.466856 30291 raft_consensus.cc:348] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac4e9ddb8a64f96b294152eead317c1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40513 } }
I20260808 15:54:30.467308 30291 raft_consensus.cc:374] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260808 15:54:30.467418 30291 raft_consensus.cc:729] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dac4e9ddb8a64f96b294152eead317c1, State: Initialized, Role: FOLLOWER
I20260808 15:54:30.468189 30291 consensus_queue.cc:260] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac4e9ddb8a64f96b294152eead317c1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40513 } }
I20260808 15:54:30.468470 30291 raft_consensus.cc:388] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260808 15:54:30.468621 30291 raft_consensus.cc:482] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260808 15:54:30.468789 30291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:30.471139 30291 raft_consensus.cc:504] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac4e9ddb8a64f96b294152eead317c1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40513 } }
I20260808 15:54:30.471661 30291 leader_election.cc:302] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dac4e9ddb8a64f96b294152eead317c1; no voters:
I20260808 15:54:30.472283 30291 leader_election.cc:288] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260808 15:54:30.472401 30304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:30.473206 30304 raft_consensus.cc:686] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [term 1 LEADER]: Becoming Leader. State: Replica: dac4e9ddb8a64f96b294152eead317c1, State: Running, Role: LEADER
I20260808 15:54:30.473793 30304 consensus_queue.cc:237] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac4e9ddb8a64f96b294152eead317c1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40513 } }
I20260808 15:54:30.474784 30291 sys_catalog.cc:564] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:30.477115 30305 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dac4e9ddb8a64f96b294152eead317c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac4e9ddb8a64f96b294152eead317c1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40513 } } }
I20260808 15:54:30.477797 30305 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:30.477545 30306 sys_catalog.cc:455] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dac4e9ddb8a64f96b294152eead317c1. Latest consensus state: current_term: 1 leader_uuid: "dac4e9ddb8a64f96b294152eead317c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dac4e9ddb8a64f96b294152eead317c1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 40513 } } }
I20260808 15:54:30.478138 30306 sys_catalog.cc:458] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:30.481451 30316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:30.487228 30316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260808 15:54:30.495983 30316 catalog_manager.cc:1347] Generated new cluster ID: 1046ba207ba74339867a77c58f5e1a80
I20260808 15:54:30.496145 30316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:30.507463 30316 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:30.507915 30316 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:30.517879 30316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dac4e9ddb8a64f96b294152eead317c1: Generated new TSK 0
I20260808 15:54:30.518716 30316 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260808 15:54:30.527352 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30193
W20260808 15:54:30.528515 30286 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50637
W20260808 15:54:30.529238 27987 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30193: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260808 15:54:30.529389 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30193
2026-08-08T15:54:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2277 ms)
[----------] 18 tests from AuthTokenIssuingTest (42259 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:30 dist-test-slave-twwt krb5kdc[30335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:30 dist-test-slave-twwt krb5kdc[30335](info): set up 2 sockets
Aug 08 15:54:30 dist-test-slave-twwt krb5kdc[30335](info): commencing operation
krb5kdc: starting...
W20260808 15:54:31.905694 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:32 dist-test-slave-twwt krb5kdc[30335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:32.284171 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42339
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42339,127.27.84.253:41993,127.27.84.252:36031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:32.645150 30358 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:54:32.645525 30358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:32.645663 30358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:32.645810 30358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:32.653072 30358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:32.653160 30358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:32.653267 30358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:32.653362 30358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:32.653460 30358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:32.663903 30358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42339,127.27.84.253:41993,127.27.84.252:36031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:32.665236 30358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:32.666684 30358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:32.678984 30378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:32.679057 30367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:32.679025 30368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:32.679880 30358 server_base.cc:1034] running on GCE node
I20260808 15:54:32.680629 30358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:32.682076 30358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:32.683295 30358 hybrid_clock.cc:648] HybridClock initialized: now 1786204472683220 us; error 47 us; skew 500 ppm
I20260808 15:54:32.683634 30358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:32 dist-test-slave-twwt krb5kdc[30335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:32.691166 30358 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:32.692746 30358 webserver.cc:460] Webserver started at http://127.27.84.254:37627/ using document root <none> and password file <none>
I20260808 15:54:32.693468 30358 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:32.693604 30358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:32.693943 30358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:32.697019 30358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "503a5a4b2f5242849adeabaa17d172a1"
format_stamp: "Formatted at 2026-08-08 15:54:32 on dist-test-slave-twwt"
server_key: "5ae559c6212ab452e80358ed4546ae4d"
server_key_iv: "9680fae828fa0a6a0cdbc9602e2b0356"
server_key_version: "encryptionkey@0"
I20260808 15:54:32.697861 30358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "503a5a4b2f5242849adeabaa17d172a1"
format_stamp: "Formatted at 2026-08-08 15:54:32 on dist-test-slave-twwt"
server_key: "5ae559c6212ab452e80358ed4546ae4d"
server_key_iv: "9680fae828fa0a6a0cdbc9602e2b0356"
server_key_version: "encryptionkey@0"
I20260808 15:54:32.703145 30358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260808 15:54:32.706759 30391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:32.708114 30358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:54:32.708305 30358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "503a5a4b2f5242849adeabaa17d172a1"
format_stamp: "Formatted at 2026-08-08 15:54:32 on dist-test-slave-twwt"
server_key: "5ae559c6212ab452e80358ed4546ae4d"
server_key_iv: "9680fae828fa0a6a0cdbc9602e2b0356"
server_key_version: "encryptionkey@0"
I20260808 15:54:32.708542 30358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:32.740836 30358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:32.745157 30358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:32.745455 30358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:32.764981 30358 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:42339
I20260808 15:54:32.765002 30460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:42339 every 8 connection(s)
I20260808 15:54:32.766477 30358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:32.770486 30463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:32.771332 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30358
I20260808 15:54:32.771574 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:32.772253 27987 external_mini_cluster.cc:1428] Setting key 70cf73ec0b009e78c22972c76f6c8467
I20260808 15:54:32.777544 30463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:54:32 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:32.800657 30465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.780935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.27.84.254:42339)
0808 15:54:32.781815 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:32.781826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:32.782428 (+ 602us) server_negotiation.cc:407] Connection header received
0808 15:54:32.784495 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:32.784500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:32.784719 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:32.784932 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:32.786242 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:32.787499 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:32.788881 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:32.789442 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:32.789644 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:32.793253 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:32.793261 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:32.793274 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:32.793300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:32.796103 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:32.796785 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:32.796800 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:32.796802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:32.796953 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:32.797357 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:32.797363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:32.797366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:32.797947 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:32.798161 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:32.798734 (+ 573us) server_negotiation.cc:299] Negotiation successful
0808 15:54:32.799161 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":681,"thread_start_us":460,"threads_started":1}
I20260808 15:54:32.800678 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.780941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42339 (local address 127.0.0.1:43200)
0808 15:54:32.782257 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.782296 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:32.784113 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:32.784952 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:32.784958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:32.785685 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:32.786025 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:32.786042 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:32.787656 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:32.787662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:32.788733 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:32.788745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:32.789686 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:32.789693 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:32.789871 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:32.790545 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:32.790567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:32.793031 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:32.796234 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:32.796246 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:32.796258 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:32.796652 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:32.797074 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:32.797079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:32.797082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:32.797245 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:32.798103 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:32.798116 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:32.798335 (+ 219us) client_negotiation.cc:769] Sending connection context
0808 15:54:32.798596 (+ 261us) client_negotiation.cc:240] Negotiation successful
0808 15:54:32.799280 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":195,"thread_start_us":926,"threads_started":1}
I20260808 15:54:32.804136 30463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:32.805269 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.804866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49568)
0808 15:54:32.805039 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.805149 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 15:54:32.805521 30396 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.84.253@KRBTEST.COM; defaulting to no policy
I20260808 15:54:32.807265 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.806936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49574)
0808 15:54:32.807077 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.807146 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:54:32.807595 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41993: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab.
Entry for principal kudu/127.27.84.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab.
I20260808 15:54:32.850224 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:41993
--webserver_interface=127.27.84.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab
--principal=kudu/127.27.84.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42339,127.27.84.253:41993,127.27.84.252:36031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:54:32.856994 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } attempt: 1
I20260808 15:54:32.858326 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.857714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49584)
0808 15:54:32.857976 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.858125 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 15:54:32.859489 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.859030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49592)
0808 15:54:32.859211 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.859305 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:54:32.859706 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41993: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
I20260808 15:54:32.927965 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } attempt: 2
I20260808 15:54:32.929303 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.928741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49600)
0808 15:54:32.928989 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.929084 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260808 15:54:32.930447 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:32.930098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49608)
0808 15:54:32.930232 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:32.930311 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 15:54:32.930727 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41993: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
I20260808 15:54:33.022010 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } attempt: 3
I20260808 15:54:33.023272 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.022823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49618)
0808 15:54:33.023031 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.023129 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260808 15:54:33.024498 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.024118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49628)
0808 15:54:33.024276 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.024352 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:54:33.024824 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41993: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
I20260808 15:54:33.168141 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } attempt: 4
I20260808 15:54:33.169571 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.168989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49642)
0808 15:54:33.169334 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.169428 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260808 15:54:33.170885 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.170379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49652)
0808 15:54:33.170564 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.170735 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260808 15:54:33.171114 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:41993: Network error: Client connection negotiation failed: client connection to 127.27.84.253:41993: connect: Connection refused (error 111)
W20260808 15:54:33.209172 30475 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:54:33.209581 30475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:33.209704 30475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:33.209846 30475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:33.217634 30475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:33.217749 30475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:33.217860 30475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:33.217957 30475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:33.218078 30475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:33.228556 30475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42339,127.27.84.253:41993,127.27.84.252:36031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab
--principal=kudu/127.27.84.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:41993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.84.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:33.230058 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:33.231547 30475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:33.244309 30481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:33.244225 30485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:33.244267 30480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:33.245313 30475 server_base.cc:1034] running on GCE node
I20260808 15:54:33.246127 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:33.247766 30475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:33.249022 30475 hybrid_clock.cc:648] HybridClock initialized: now 1786204473248952 us; error 48 us; skew 500 ppm
I20260808 15:54:33.249348 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:33.252823 30475 init.cc:377] Logged in from keytab as kudu/127.27.84.253@KRBTEST.COM (short username kudu)
I20260808 15:54:33.254267 30475 webserver.cc:460] Webserver started at http://127.27.84.253:41863/ using document root <none> and password file <none>
I20260808 15:54:33.255035 30475 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:33.255126 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:33.255493 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:33.258553 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data/instance:
uuid: "2f93320d5be543358d7c8da7386c4e4d"
format_stamp: "Formatted at 2026-08-08 15:54:33 on dist-test-slave-twwt"
server_key: "ddd733ad4b3adf09e13b3d853d43a6fd"
server_key_iv: "75e2b9c3dc2f6ec8599013e364409f1d"
server_key_version: "encryptionkey@0"
I20260808 15:54:33.259419 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal/instance:
uuid: "2f93320d5be543358d7c8da7386c4e4d"
format_stamp: "Formatted at 2026-08-08 15:54:33 on dist-test-slave-twwt"
server_key: "ddd733ad4b3adf09e13b3d853d43a6fd"
server_key_iv: "75e2b9c3dc2f6ec8599013e364409f1d"
server_key_version: "encryptionkey@0"
I20260808 15:54:33.264659 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:54:33.268232 30495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:33.269732 30475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260808 15:54:33.269922 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal
uuid: "2f93320d5be543358d7c8da7386c4e4d"
format_stamp: "Formatted at 2026-08-08 15:54:33 on dist-test-slave-twwt"
server_key: "ddd733ad4b3adf09e13b3d853d43a6fd"
server_key_iv: "75e2b9c3dc2f6ec8599013e364409f1d"
server_key_version: "encryptionkey@0"
I20260808 15:54:33.270138 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:33.297452 30475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:33.301898 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:33.302205 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:33.321674 30475 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.253:41993
I20260808 15:54:33.321710 30571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.253:41993 every 8 connection(s)
I20260808 15:54:33.323191 30475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
I20260808 15:54:33.326874 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30475
I20260808 15:54:33.327059 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-1/wal/instance
I20260808 15:54:33.327199 30572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:33.327574 27987 external_mini_cluster.cc:1428] Setting key f7fd19876110f523cb1117af17698cd7
I20260808 15:54:33.334035 30572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:33.355756 30576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.336783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.27.84.254:42339)
0808 15:54:33.337346 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.337351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.338319 (+ 968us) server_negotiation.cc:407] Connection header received
0808 15:54:33.340236 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.340245 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.340340 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.340466 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.342042 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.342909 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.344431 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.344851 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.345004 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.349194 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.349200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.349203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.349241 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.352086 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.353215 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.353220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.353222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.353342 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.353863 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.353866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.353868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.354156 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.354279 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.355309 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.355479 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260808 15:54:33.357575 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.337379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42339 (local address 127.0.0.1:43216)
0808 15:54:33.338153 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.338191 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.339960 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.340721 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.340729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.341246 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.341813 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.341830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.343086 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.343091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.344266 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.344279 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.345091 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.345099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.345338 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.346036 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.346058 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.348856 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.352256 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.352269 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.352291 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.353092 (+ 801us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.353462 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.353471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.353478 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.353748 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.354303 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.354315 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.354890 (+ 575us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.355246 (+ 356us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.355923 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":334,"threads_started":1}
I20260808 15:54:33.359457 30572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.84.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.252@KRBTEST.COM" created.
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:33.377800 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.360336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49658)
0808 15:54:33.360538 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.360568 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.360654 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.361821 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.361831 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.362426 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.362707 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.362724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.364183 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.364191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.365839 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.365861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.366792 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.366798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.367027 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.367577 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.367594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.370135 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.374765 (+ 4630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.374769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.374772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.375297 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.375763 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.375770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.375776 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.375937 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.376702 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.376708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.376867 (+ 159us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.377006 (+ 139us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.377291 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":6}
I20260808 15:54:33.378175 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.360600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.27.84.253:41993)
0808 15:54:33.361346 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.361351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.361377 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:54:33.361424 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.361429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.361601 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.361721 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.362910 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.364004 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.366006 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.366564 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.366781 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.370373 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.370409 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.370431 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.370476 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.374587 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.375439 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.375451 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.375456 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.375624 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.376051 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.376058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.376064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.376538 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.376766 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.377190 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.377475 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":590,"thread_start_us":453,"threads_started":1}
I20260808 15:54:33.380846 30572 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:33.382107 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.381535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33124)
0808 15:54:33.381781 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.381928 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260808 15:54:33.382396 30502 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.384121 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.383682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33130)
0808 15:54:33.383859 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.383976 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260808 15:54:33.384338 30572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab.
Entry for principal kudu/127.27.84.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab.
I20260808 15:54:33.410554 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:36031
--webserver_interface=127.27.84.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab
--principal=kudu/127.27.84.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42339,127.27.84.253:41993,127.27.84.252:36031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:54:33.433677 30572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 1
I20260808 15:54:33.434947 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.434406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33134)
0808 15:54:33.434657 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.434768 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260808 15:54:33.436005 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.435660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33138)
0808 15:54:33.435824 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.435895 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:54:33.436208 30572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.470389 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } attempt: 5
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:33.486315 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.471332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.27.84.253:41993)
0808 15:54:33.471565 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.471571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.471596 (+ 25us) server_negotiation.cc:407] Connection header received
0808 15:54:33.471727 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.471734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.471838 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.471968 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.472796 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.474105 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.476193 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.476636 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.476743 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.480789 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.480795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.480798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.480832 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.483604 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.484300 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.484310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.484312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.484436 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.484828 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.484833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.484836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.485139 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.485304 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.485728 (+ 424us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.485964 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260808 15:54:33.486306 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.471228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49672)
0808 15:54:33.471431 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.471456 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.471583 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.472019 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.472022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.472409 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.472594 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.472602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.474362 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.474371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.476009 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.476031 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.476863 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.476873 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.477193 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.477933 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.477957 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.480575 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.483754 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.483760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.483764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.484157 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.484554 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.484559 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.484563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.484710 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.485273 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.485278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.485453 (+ 175us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.485614 (+ 161us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.485952 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 15:54:33.487424 30463 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:33.488557 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.488053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33144)
0808 15:54:33.488261 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.488372 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260808 15:54:33.489529 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.489188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33156)
0808 15:54:33.489345 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.489413 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:54:33.489817 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.504431 30572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 2
I20260808 15:54:33.505564 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.505090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33166)
0808 15:54:33.505306 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.505404 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260808 15:54:33.506825 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.506380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33170)
0808 15:54:33.506572 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.506703 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260808 15:54:33.507100 30572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.541080 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 1
I20260808 15:54:33.542320 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.541841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33176)
0808 15:54:33.542054 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.542148 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260808 15:54:33.543421 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.543071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33184)
0808 15:54:33.543237 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.543311 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260808 15:54:33.543671 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.598385 30572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 3
I20260808 15:54:33.599797 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.599220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33188)
0808 15:54:33.599460 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.599556 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260808 15:54:33.600893 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.600475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33196)
0808 15:54:33.600699 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.600780 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260808 15:54:33.601150 30572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.611949 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 2
I20260808 15:54:33.613257 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.612734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33200)
0808 15:54:33.612960 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.613051 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260808 15:54:33.614234 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.613932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33210)
0808 15:54:33.614065 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.614132 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:54:33.614471 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.720695 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 3
I20260808 15:54:33.721922 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.721442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33214)
0808 15:54:33.721651 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.721745 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 15:54:33.723049 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.722588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33230)
0808 15:54:33.722869 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.722945 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260808 15:54:33.723245 30463 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
I20260808 15:54:33.744392 30572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 4
I20260808 15:54:33.745769 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.745213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33244)
0808 15:54:33.745486 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.745586 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260808 15:54:33.746845 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.746454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33260)
0808 15:54:33.746651 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.746732 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260808 15:54:33.747063 30572 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36031: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36031: connect: Connection refused (error 111)
W20260808 15:54:33.778122 30586 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:54:33.778513 30586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:33.778671 30586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:33.778825 30586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:33.786379 30586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:33.786509 30586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:33.786674 30586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:33.786784 30586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:33.786880 30586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:33.797160 30586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42339,127.27.84.253:41993,127.27.84.252:36031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab
--principal=kudu/127.27.84.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.84.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:33.798642 30586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:33.800101 30586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:33.812521 30594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:33.812664 30596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:33.812636 30593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:33.813275 30586 server_base.cc:1034] running on GCE node
I20260808 15:54:33.814239 30586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:33.815894 30586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:33.817226 30586 hybrid_clock.cc:648] HybridClock initialized: now 1786204473817155 us; error 45 us; skew 500 ppm
I20260808 15:54:33.817624 30586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:33.821165 30586 init.cc:377] Logged in from keytab as kudu/127.27.84.252@KRBTEST.COM (short username kudu)
I20260808 15:54:33.822875 30586 webserver.cc:460] Webserver started at http://127.27.84.252:34597/ using document root <none> and password file <none>
I20260808 15:54:33.823581 30586 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:33.823670 30586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:33.823958 30586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:33.827092 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data/instance:
uuid: "5a1cb45097cf401db345b00b4f7b058b"
format_stamp: "Formatted at 2026-08-08 15:54:33 on dist-test-slave-twwt"
server_key: "ce76d074bd954a1cf05a77f6c0493773"
server_key_iv: "c35a21f94ea98eed0168d2352000394a"
server_key_version: "encryptionkey@0"
I20260808 15:54:33.827859 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal/instance:
uuid: "5a1cb45097cf401db345b00b4f7b058b"
format_stamp: "Formatted at 2026-08-08 15:54:33 on dist-test-slave-twwt"
server_key: "ce76d074bd954a1cf05a77f6c0493773"
server_key_iv: "c35a21f94ea98eed0168d2352000394a"
server_key_version: "encryptionkey@0"
I20260808 15:54:33.833293 30586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260808 15:54:33.836928 30609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:33.838346 30586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260808 15:54:33.838541 30586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal
uuid: "5a1cb45097cf401db345b00b4f7b058b"
format_stamp: "Formatted at 2026-08-08 15:54:33 on dist-test-slave-twwt"
server_key: "ce76d074bd954a1cf05a77f6c0493773"
server_key_iv: "c35a21f94ea98eed0168d2352000394a"
server_key_version: "encryptionkey@0"
I20260808 15:54:33.838907 30586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:33.853078 30586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:33.857532 30586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:33.857824 30586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:33.878015 30586 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.252:36031
I20260808 15:54:33.878027 30689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.252:36031 every 8 connection(s)
I20260808 15:54:33.879654 30586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
I20260808 15:54:33.883675 30690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:33.886859 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30586
I20260808 15:54:33.887069 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786204089698720-27987-0/minicluster-data/master-2/wal/instance
I20260808 15:54:33.887534 27987 external_mini_cluster.cc:1428] Setting key e45cfa5e97bf6036da705ddcea631d59
I20260808 15:54:33.891726 30690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:33.900458 30463 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 4
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:54:33.913697 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.895150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.27.84.254:42339)
0808 15:54:33.895763 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.895769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.896377 (+ 608us) server_negotiation.cc:407] Connection header received
0808 15:54:33.896602 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.896607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.896671 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.896754 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.898178 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.899253 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.901228 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.901975 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.902122 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.906473 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.906483 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.906488 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.906540 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.910128 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.910989 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.910998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.911004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.911151 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.911692 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.911699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.911702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.912059 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.912204 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.913072 (+ 868us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.913294 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":285,"threads_started":1}
I20260808 15:54:33.914999 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.895204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42339 (local address 127.0.0.1:43226)
0808 15:54:33.896203 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.896252 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.896453 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.896852 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.896859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.897749 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.898011 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.898020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.899406 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.899411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.901018 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.901033 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.902278 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.902295 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.902421 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.903303 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.903334 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.906277 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.910295 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.910309 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.910327 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.910870 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.911286 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.911293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.911299 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.911563 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.912279 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.912290 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.912591 (+ 301us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.912967 (+ 376us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.913520 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":480,"threads_started":1}
I20260808 15:54:33.915053 30695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.889590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.27.84.252:36031)
0808 15:54:33.890438 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.890451 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.890498 (+ 47us) server_negotiation.cc:407] Connection header received
0808 15:54:33.892358 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.892383 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.892861 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.893298 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.895182 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.897415 (+ 2233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.899198 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.900292 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.900537 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.905969 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.905982 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.906004 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.906048 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.909901 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.910871 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.910880 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.910883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.911097 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.911519 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.911524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.911527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.912400 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.912753 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.914275 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.914538 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":391,"threads_started":1}
I20260808 15:54:33.917371 30690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:33.919000 30466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.901100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33280)
0808 15:54:33.901300 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.901328 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.901409 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.902722 (+ 1313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.902729 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.903260 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.903473 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.903487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.904899 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.904906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.906026 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.906041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.907008 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.907015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.907247 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.907932 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.907961 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.910948 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.915476 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.915481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.915484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.915996 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.917241 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.917245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.917249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.917407 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.918062 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.918067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.918236 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.918401 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.918707 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260808 15:54:33.919903 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.901595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.27.84.252:36031)
0808 15:54:33.902314 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.902321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.902347 (+ 26us) server_negotiation.cc:407] Connection header received
0808 15:54:33.902400 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.902405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.902493 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.902658 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.903668 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.904752 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.906187 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.906846 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.906978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.911212 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.911221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.911228 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.911267 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.915313 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.916447 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.916454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.916459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.916645 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.917545 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.917550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.917553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.917932 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.918091 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.919418 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.919594 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":388,"threads_started":1}
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:33.926469 30463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: Bootstrap starting.
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:33.930799 30463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:33.932255 30463 log.cc:824] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:33.932374 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.918091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:41993 (local address 127.0.0.1:49676)
0808 15:54:33.918242 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.918263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.918362 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.919004 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.919008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.919489 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.919722 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.919734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.921265 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.921270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.922127 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.922138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.922894 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.922900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.923099 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.923683 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.923700 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.925820 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.929300 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.929308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.929313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.929871 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.930396 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.930401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.930404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.930595 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.931361 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.931366 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.931528 (+ 162us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.931687 (+ 159us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.931991 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260808 15:54:33.932549 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.918502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.27.84.253:41993)
0808 15:54:33.918709 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.918717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.918736 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:54:33.918798 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.918803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.918885 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.918978 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.919897 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.920868 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.922272 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.922746 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.922888 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.926023 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.926029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.926033 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.926065 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.929151 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.930057 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.930063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.930066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.930238 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.930767 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.930774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.930778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.931195 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.931352 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.931825 (+ 473us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.932122 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260808 15:54:33.934006 30690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:33.937691 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.919222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.27.84.254:42339)
0808 15:54:33.919401 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.919406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.919425 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:54:33.919474 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.919478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.919558 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.921104 (+ 1546us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.921239 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.922269 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.924142 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.924774 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.925497 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.929985 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.929990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.929993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.930054 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.933563 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.934815 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.934823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.934830 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.935032 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.936025 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.936030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.936033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.936418 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.936726 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.937149 (+ 423us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.937378 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260808 15:54:33.942395 30463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: No bootstrap required, opened a new log
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
Aug 08 15:54:33 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:33.952998 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.934910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.27.84.252:36031)
0808 15:54:33.935079 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.935084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.935424 (+ 340us) server_negotiation.cc:407] Connection header received
0808 15:54:33.935589 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.935592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.935662 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.935769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.936658 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.937656 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.939234 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.939782 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.939908 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.944004 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.944011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.944016 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.944057 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.947805 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.948713 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.948719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.948723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.948876 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.949324 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.949330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.949333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.949762 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.949946 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.951475 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.951843 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:54:33.953178 30463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
I20260808 15:54:33.953616 30463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 503a5a4b2f5242849adeabaa17d172a1, State: Initialized, Role: FOLLOWER
I20260808 15:54:33.954280 30701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.934733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33284)
0808 15:54:33.935318 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:33.935341 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:33.935483 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:33.935789 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:33.935793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:33.936292 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:33.936509 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.936522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.937823 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.937831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.939048 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:33.939060 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.940072 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.940076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:33.940193 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.940775 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:33.940802 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:33.943790 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:33.947964 (+ 4174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.947973 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.947978 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.948572 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.949014 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:33.949020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:33.949024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:33.949191 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:33.950734 (+ 1543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:33.950740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:33.951010 (+ 270us) client_negotiation.cc:769] Sending connection context
0808 15:54:33.951273 (+ 263us) client_negotiation.cc:240] Negotiation successful
0808 15:54:33.952736 (+ 1463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421}
I20260808 15:54:33.954198 30463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
I20260808 15:54:33.958890 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:33.939922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.27.84.253:41993)
0808 15:54:33.940095 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:33.940102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:33.940120 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:54:33.940281 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:33.940289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:33.940369 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:33.940477 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:33.942148 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.943260 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.944735 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:33.945307 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:33.945469 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:33.950421 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:33.950431 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:33.950440 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:33.950490 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:33.954649 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.955609 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.955617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.955621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.955819 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:33.956331 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:33.956339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:33.956344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:33.956715 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:33.956999 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:33.958281 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0808 15:54:33.958536 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:54:33.961809 30690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: Bootstrap starting.
I20260808 15:54:33.961892 30704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:33.962357 30704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:33.964188 30463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:33.965915 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:33.966123 30690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:33.967470 30690 log.cc:824] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:33.970124 30425 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "503a5a4b2f5242849adeabaa17d172a1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43240
I20260808 15:54:33.971341 30425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 15:54:33.971597 30425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:33.974157 30690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: No bootstrap required, opened a new log
I20260808 15:54:33.975468 30425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
W20260808 15:54:33.975764 30722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:54:33.975884 30722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:54:33.977730 30425 leader_election.cc:288] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [CANDIDATE]: Term 1 election: Requested vote from peers 2f93320d5be543358d7c8da7386c4e4d (127.27.84.253:41993), 5a1cb45097cf401db345b00b4f7b058b (127.27.84.252:36031)
I20260808 15:54:33.990902 30690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
I20260808 15:54:33.991694 30690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1cb45097cf401db345b00b4f7b058b, State: Initialized, Role: FOLLOWER
I20260808 15:54:33.992486 30690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
W20260808 15:54:33.998507 30398 tablet.cc:2369] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:54:33.981765 30358 master_runner.cc:186] Error getting master registration for 127.27.84.253:41993: OK, instance_id { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" instance_seqno: 1786204473316060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:54:34.006436 30358 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:54:34.008368 30726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.009425 30726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:34.009577 30690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:34.009675 30647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "503a5a4b2f5242849adeabaa17d172a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5a1cb45097cf401db345b00b4f7b058b"
I20260808 15:54:34.010394 30647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:34.014554 30647 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 503a5a4b2f5242849adeabaa17d172a1 in term 1.
I20260808 15:54:34.015676 30395 leader_election.cc:302] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 503a5a4b2f5242849adeabaa17d172a1, 5a1cb45097cf401db345b00b4f7b058b; no voters:
I20260808 15:54:34.016284 30704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260808 15:54:34.026679 30704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [term 1 LEADER]: Becoming Leader. State: Replica: 503a5a4b2f5242849adeabaa17d172a1, State: Running, Role: LEADER
I20260808 15:54:34.027552 30704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
I20260808 15:54:34.039299 30733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503a5a4b2f5242849adeabaa17d172a1. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.039927 30733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:34.046262 30572 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } attempt: 5
I20260808 15:54:34.048656 30741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:34.052424 30741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 08 15:54:34 dist-test-slave-twwt krb5kdc[30335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:54:34.081365 30577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:34.047396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36031 (local address 127.0.0.1:33300)
0808 15:54:34.047664 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:34.047688 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:34.047844 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:34.051134 (+ 3290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:34.051142 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:34.051770 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:34.052023 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:34.052039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.054550 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.054558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:34.056142 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:34.056159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.057649 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.057660 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:34.058020 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:34.059029 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:34.059068 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:34.066836 (+ 7768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:34.075620 (+ 8784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:34.075627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:34.075631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:34.076443 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:34.077527 (+ 1084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:34.077535 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:34.077541 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:34.077784 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:34.079730 (+ 1946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:34.079739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:34.079998 (+ 259us) client_negotiation.cc:769] Sending connection context
0808 15:54:34.080195 (+ 197us) client_negotiation.cc:240] Negotiation successful
0808 15:54:34.080687 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260808 15:54:34.084949 30763 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260808 15:54:34.087515 30698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:34.049842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.27.84.254:42339)
0808 15:54:34.050302 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.050311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.053919 (+ 3608us) server_negotiation.cc:407] Connection header received
0808 15:54:34.054487 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.054497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.054626 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.054715 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.059815 (+ 5100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.061332 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.066285 (+ 4953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.067381 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.067512 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:34.075567 (+ 8055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:34.075578 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:34.075585 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:34.075645 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:34.080922 (+ 5277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.083055 (+ 2133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.083067 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.083073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.083538 (+ 465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.084584 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.084594 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.084600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.085366 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:34.085619 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:34.086352 (+ 733us) server_negotiation.cc:299] Negotiation successful
0808 15:54:34.086799 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260808 15:54:34.088526 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:34.050049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.27.84.252:36031)
0808 15:54:34.050477 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.050484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.050506 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:54:34.050637 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.050646 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.050753 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.050891 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.052838 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.054085 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.056366 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.057114 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.057304 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:34.068358 (+ 11054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:34.068370 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:34.068379 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:34.068442 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:34.075420 (+ 6978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.077052 (+ 1632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.077063 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.077071 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.077337 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.078874 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.078883 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.078890 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.079482 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:34.086028 (+ 6546us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:34.086221 (+ 193us) server_negotiation.cc:299] Negotiation successful
0808 15:54:34.086501 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260808 15:54:34.096472 30572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: Bootstrap starting.
I20260808 15:54:34.102666 30572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:34.092615 30586 master_runner.cc:186] Error getting master registration for 127.27.84.253:41993: OK, instance_id { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" instance_seqno: 1786204473316060 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:54:34.103749 30586 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260808 15:54:34.109944 30616 tablet.cc:2369] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:54:34.116792 30572 log.cc:824] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:34.114248 30579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:34.052409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49696 (local address 127.27.84.253:41993)
0808 15:54:34.052591 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.052597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.086882 (+ 34285us) server_negotiation.cc:407] Connection header received
0808 15:54:34.087170 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.087176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.087257 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.087328 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.089257 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.090373 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.092143 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.092892 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.093018 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:34.106389 (+ 13371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:34.106396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:34.106399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:34.106439 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:34.110162 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.111555 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.111561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.111565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.111744 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.112297 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.112303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.112310 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.112806 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:34.113077 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:34.113638 (+ 561us) server_negotiation.cc:299] Negotiation successful
0808 15:54:34.113890 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260808 15:54:34.125717 30572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: No bootstrap required, opened a new log
I20260808 15:54:34.128934 30647 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [term 1 FOLLOWER]: Refusing update from remote peer 503a5a4b2f5242849adeabaa17d172a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:34.130054 30733 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:34.137751 30572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
I20260808 15:54:34.138339 30572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f93320d5be543358d7c8da7386c4e4d, State: Initialized, Role: FOLLOWER
I20260808 15:54:34.138986 30572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } }
I20260808 15:54:34.141848 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:34.112272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.27.84.252:36031)
0808 15:54:34.112605 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.112611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.114395 (+ 1784us) server_negotiation.cc:407] Connection header received
0808 15:54:34.114466 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.114472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.114552 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.114683 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.116343 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.117540 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.119461 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:34.120140 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:34.120261 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:34.123417 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:34.123424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:34.123429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:34.123472 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:34.127132 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.137075 (+ 9943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.137080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.137083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.137292 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:34.137955 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:34.137960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:34.137963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:34.138401 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:34.138698 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:34.140966 (+ 2268us) server_negotiation.cc:299] Negotiation successful
0808 15:54:34.141317 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260808 15:54:34.143599 30768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.144104 30768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:34.144886 30572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:34.145285 30538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:34.150149 30538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [term 1 FOLLOWER]: Refusing update from remote peer 503a5a4b2f5242849adeabaa17d172a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:34.150805 30542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "503a5a4b2f5242849adeabaa17d172a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f93320d5be543358d7c8da7386c4e4d"
I20260808 15:54:34.153177 30726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503a5a4b2f5242849adeabaa17d172a1. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.153512 30726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:34.155665 30542 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 503a5a4b2f5242849adeabaa17d172a1 in term 1.
I20260808 15:54:34.156615 30754 mvcc.cc:205] Tried to move back new op lower bound from 7316293526000226304 to 7316293525624557568. Current Snapshot: MvccSnapshot[applied={T|T < 7316293526000226304}]
I20260808 15:54:34.157444 30704 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:34.161489 30766 mvcc.cc:205] Tried to move back new op lower bound from 7316293526000226304 to 7316293525624557568. Current Snapshot: MvccSnapshot[applied={T|T < 7316293526000226304}]
I20260808 15:54:34.165933 30704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.166316 30704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:34.171130 30741 catalog_manager.cc:1347] Generated new cluster ID: 0db1dbd2beca4c5599e184c84aaa1f5e
I20260808 15:54:34.171290 30741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:34.176792 30726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.177174 30726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:34.189960 30768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 503a5a4b2f5242849adeabaa17d172a1. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.190426 30768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:34.191406 30785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.191931 30785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:34.222393 30801 catalog_manager.cc:1259] Loaded cluster ID: 0db1dbd2beca4c5599e184c84aaa1f5e
I20260808 15:54:34.222649 30801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: loading cluster ID for follower catalog manager: success
W20260808 15:54:34.224115 30801 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260808 15:54:34.224373 30801 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:54:34.242781 30733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "503a5a4b2f5242849adeabaa17d172a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "503a5a4b2f5242849adeabaa17d172a1" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42339 } } peers { permanent_uuid: "2f93320d5be543358d7c8da7386c4e4d" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 41993 } } peers { permanent_uuid: "5a1cb45097cf401db345b00b4f7b058b" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36031 } } }
I20260808 15:54:34.243161 30733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1 [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:34.246726 30741 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:34.248217 30741 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:34.273623 30741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 503a5a4b2f5242849adeabaa17d172a1: Generated new TSK 0
I20260808 15:54:34.275236 30741 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:54:34.385898 30698 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.370587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.27.84.254:42339)
0808 15:54:34.370837 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.370843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.382519 (+ 11676us) server_negotiation.cc:407] Connection header received
0808 15:54:34.384113 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.384121 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.384210 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.384286 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.385603 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43266: BlockingRecv error: recv got EOF from 127.0.0.1:43266 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 15:54:34.389199 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.370197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.27.84.253:41993)
0808 15:54:34.370426 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.370434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.387290 (+ 16856us) server_negotiation.cc:407] Connection header received
0808 15:54:34.387612 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.387617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.387703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.387820 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.388973 (+ 1153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49712: BlockingRecv error: recv got EOF from 127.0.0.1:49712 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:54:34.389326 30702 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.383237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.27.84.252:36031)
0808 15:54:34.383430 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.383436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.387444 (+ 4008us) server_negotiation.cc:407] Connection header received
0808 15:54:34.387985 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.387991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.388078 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.388149 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.389044 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33324: BlockingRecv error: recv got EOF from 127.0.0.1:33324 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 15:54:34.398341 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.395387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49720 (local address 127.27.84.253:41993)
0808 15:54:34.395581 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.395587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.396089 (+ 502us) server_negotiation.cc:407] Connection header received
0808 15:54:34.396313 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.396319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.396447 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.397615 (+ 1168us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.398035 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49720: BlockingRecv error: recv got EOF from 127.0.0.1:49720 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260808 15:54:34.407239 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.404951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.27.84.253:41993)
0808 15:54:34.405137 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.405143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.405700 (+ 557us) server_negotiation.cc:407] Connection header received
0808 15:54:34.405906 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.405911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.406017 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.406137 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.407014 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49724: BlockingRecv error: recv got EOF from 127.0.0.1:49724 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260808 15:54:34.421305 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.418862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.27.84.253:41993)
0808 15:54:34.419066 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.419072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.419608 (+ 536us) server_negotiation.cc:407] Connection header received
0808 15:54:34.419856 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.419863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.419988 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.420105 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.421039 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49728: BlockingRecv error: recv got EOF from 127.0.0.1:49728 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260808 15:54:34.431056 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.428664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49736 (local address 127.27.84.253:41993)
0808 15:54:34.428918 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.428924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.429471 (+ 547us) server_negotiation.cc:407] Connection header received
0808 15:54:34.429653 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.429660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.429770 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.429896 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.430813 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49736: BlockingRecv error: recv got EOF from 127.0.0.1:49736 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
W20260808 15:54:34.445462 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.442933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.27.84.253:41993)
0808 15:54:34.443147 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.443153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.443858 (+ 705us) server_negotiation.cc:407] Connection header received
0808 15:54:34.444099 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.444105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.444213 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.444337 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.445206 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49746: BlockingRecv error: recv got EOF from 127.0.0.1:49746 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260808 15:54:34.469579 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.466973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.27.84.253:41993)
0808 15:54:34.467160 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.467166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.467717 (+ 551us) server_negotiation.cc:407] Connection header received
0808 15:54:34.468080 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.468087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.468181 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.468255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.469299 (+ 1044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49758: BlockingRecv error: recv got EOF from 127.0.0.1:49758 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260808 15:54:34.515547 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.512861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.27.84.253:41993)
0808 15:54:34.513152 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.513159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.513854 (+ 695us) server_negotiation.cc:407] Connection header received
0808 15:54:34.514107 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.514113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.514212 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.514311 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.515272 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49774: BlockingRecv error: recv got EOF from 127.0.0.1:49774 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260808 15:54:34.585256 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.583068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.27.84.253:41993)
0808 15:54:34.583365 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.583372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.583734 (+ 362us) server_negotiation.cc:407] Connection header received
0808 15:54:34.583916 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.583930 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.584037 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.584193 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.584950 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49790: BlockingRecv error: recv got EOF from 127.0.0.1:49790 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260808 15:54:34.722925 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.720786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.27.84.253:41993)
0808 15:54:34.721087 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.721094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.721434 (+ 340us) server_negotiation.cc:407] Connection header received
0808 15:54:34.721686 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.721691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.721777 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.721894 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.722695 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49806: BlockingRecv error: recv got EOF from 127.0.0.1:49806 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260808 15:54:34.988962 30579 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:34.987192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.27.84.253:41993)
0808 15:54:34.987453 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:34.987458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:34.987603 (+ 145us) server_negotiation.cc:407] Connection header received
0808 15:54:34.987784 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:34.987789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:34.987864 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:34.987953 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:34.988739 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49818: BlockingRecv error: recv got EOF from 127.0.0.1:49818 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:54:35.089648 30763 catalog_manager.cc:1259] Loaded cluster ID: 0db1dbd2beca4c5599e184c84aaa1f5e
I20260808 15:54:35.089798 30763 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: loading cluster ID for follower catalog manager: success
I20260808 15:54:35.092584 30763 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: acquiring CA information for follower catalog manager: success
I20260808 15:54:35.094583 30763 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5a1cb45097cf401db345b00b4f7b058b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:54:35.228175 30801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: acquiring CA information for follower catalog manager: success
I20260808 15:54:35.230218 30801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f93320d5be543358d7c8da7386c4e4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:54:35.510447 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:35.504780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.27.84.253:41993)
0808 15:54:35.505280 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:35.505285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:35.505306 (+ 21us) server_negotiation.cc:407] Connection header received
0808 15:54:35.505520 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:35.505524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:35.505683 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:35.505806 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:35.506205 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.506986 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.508289 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.508702 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.508796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:35.509267 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:35.509797 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:35.509920 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:35.510072 (+ 152us) server_negotiation.cc:299] Negotiation successful
0808 15:54:35.510144 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
I20260808 15:54:35.523813 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:35.516943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.27.84.252:36031)
0808 15:54:35.517509 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:35.517517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:35.517539 (+ 22us) server_negotiation.cc:407] Connection header received
0808 15:54:35.517828 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:35.517832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:35.517995 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:35.518209 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:35.518709 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.519829 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.521337 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.521757 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.521863 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:35.522389 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:35.523135 (+ 746us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:35.523251 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:35.523379 (+ 128us) server_negotiation.cc:299] Negotiation successful
0808 15:54:35.523471 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":279,"threads_started":1}
I20260808 15:54:35.538638 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:35.530324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.27.84.253:41993)
0808 15:54:35.530534 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:35.530544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:35.531636 (+ 1092us) server_negotiation.cc:407] Connection header received
0808 15:54:35.531944 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:35.531949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:35.532044 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:35.532125 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:35.533011 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.533970 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.536367 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.537050 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.537194 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:35.537837 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:35.537961 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:35.538013 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:35.538214 (+ 201us) server_negotiation.cc:299] Negotiation successful
0808 15:54:35.538313 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:54:35.539325 30914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:35.531842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.27.84.252:36031)
0808 15:54:35.532013 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:35.532019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:35.532616 (+ 597us) server_negotiation.cc:407] Connection header received
0808 15:54:35.532878 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:35.532884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:35.532974 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:35.533073 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:35.533588 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.534559 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.536460 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.536987 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.537135 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:35.537759 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:35.537913 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:35.537980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:35.538920 (+ 940us) server_negotiation.cc:299] Negotiation successful
0808 15:54:35.539010 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260808 15:54:35.539590 30922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:35.530266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.27.84.254:42339)
0808 15:54:35.531018 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:35.531027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:35.531457 (+ 430us) server_negotiation.cc:407] Connection header received
0808 15:54:35.531781 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:35.531790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:35.532087 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:35.532277 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:35.533174 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.534202 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.536602 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:35.537287 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:35.537411 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:35.538066 (+ 655us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:35.538542 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:35.538705 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:35.539188 (+ 483us) server_negotiation.cc:299] Negotiation successful
0808 15:54:35.539278 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":440,"threads_started":1}
I20260808 15:54:35.545495 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30358
I20260808 15:54:35.546739 30455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:35.661420 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30358
I20260808 15:54:35.695744 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30475
I20260808 15:54:35.696899 30566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:35.806569 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30475
I20260808 15:54:35.835682 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30586
I20260808 15:54:35.837052 30682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:35.948416 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30586
2026-08-08T15:54:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5428 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 08 15:54:36 dist-test-slave-twwt krb5kdc[30939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 08 15:54:36 dist-test-slave-twwt krb5kdc[30939](info): set up 2 sockets
Aug 08 15:54:36 dist-test-slave-twwt krb5kdc[30939](info): commencing operation
krb5kdc: starting...
W20260808 15:54:37.382171 27987 mini_kdc.cc:121] Time spent starting KDC: real 1.379s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/test-user.keytab.
Aug 08 15:54:37 dist-test-slave-twwt krb5kdc[30939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-08T15:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-08T15:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.84.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
Entry for principal kudu/127.27.84.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab.
I20260808 15:54:37.750950 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42313
--webserver_interface=127.27.84.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42313,127.27.84.253:36825,127.27.84.252:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260808 15:54:38.114504 30961 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:54:38.115020 30961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:38.115145 30961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:38.115303 30961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:38.122990 30961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:38.123122 30961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:38.123225 30961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:38.123317 30961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:38.123409 30961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:38.133684 30961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42313,127.27.84.253:36825,127.27.84.252:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.254.keytab
--principal=kudu/127.27.84.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.254:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.84.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:38.135214 30961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:38.136622 30961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:38.149137 30967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:38.149187 30972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:38.149106 30968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:38.150238 30961 server_base.cc:1034] running on GCE node
I20260808 15:54:38.151187 30961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:38.152772 30961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:38.154042 30961 hybrid_clock.cc:648] HybridClock initialized: now 1786204478153960 us; error 46 us; skew 500 ppm
I20260808 15:54:38.154452 30961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:38 dist-test-slave-twwt krb5kdc[30939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:38.162755 30961 init.cc:377] Logged in from keytab as kudu/127.27.84.254@KRBTEST.COM (short username kudu)
I20260808 15:54:38.164485 30961 webserver.cc:460] Webserver started at http://127.27.84.254:39531/ using document root <none> and password file <none>
I20260808 15:54:38.165256 30961 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:38.165385 30961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:38.165750 30961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:38.168977 30961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data/instance:
uuid: "329566f2f6e847b1ba03b80f676f31cf"
format_stamp: "Formatted at 2026-08-08 15:54:38 on dist-test-slave-twwt"
server_key: "9340266fcccdb0ae59c1475c344d8098"
server_key_iv: "d05189487c1851b23e823578a35092e6"
server_key_version: "encryptionkey@0"
I20260808 15:54:38.169806 30961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal/instance:
uuid: "329566f2f6e847b1ba03b80f676f31cf"
format_stamp: "Formatted at 2026-08-08 15:54:38 on dist-test-slave-twwt"
server_key: "9340266fcccdb0ae59c1475c344d8098"
server_key_iv: "d05189487c1851b23e823578a35092e6"
server_key_version: "encryptionkey@0"
I20260808 15:54:38.175277 30961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260808 15:54:38.179044 30981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:38.180603 30961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260808 15:54:38.180752 30961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal
uuid: "329566f2f6e847b1ba03b80f676f31cf"
format_stamp: "Formatted at 2026-08-08 15:54:38 on dist-test-slave-twwt"
server_key: "9340266fcccdb0ae59c1475c344d8098"
server_key_iv: "d05189487c1851b23e823578a35092e6"
server_key_version: "encryptionkey@0"
I20260808 15:54:38.180943 30961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:38.229688 30961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:38.234220 30961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:38.234483 30961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:38.254158 30961 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.254:42313
I20260808 15:54:38.254163 31059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.254:42313 every 8 connection(s)
I20260808 15:54:38.255698 30961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/data/info.pb
I20260808 15:54:38.259430 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 30961
I20260808 15:54:38.259663 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-0/wal/instance
I20260808 15:54:38.259961 31060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:38.260190 27987 external_mini_cluster.cc:1428] Setting key b96a0c45e6e79a8473eb6d761e67aab2
I20260808 15:54:38.267273 31060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:54:38 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:38.293210 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.270536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42313 (local address 127.0.0.1:41490)
0808 15:54:38.271515 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.271573 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:38.274427 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:38.275337 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:38.275350 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:38.276203 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:38.276669 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.276688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.279101 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.279105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.280259 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.280272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.281693 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.281699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.281829 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.282541 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:38.282563 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:38.285442 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:38.288811 (+ 3369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.288827 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.288843 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.289473 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.289886 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.289893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.289899 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.290123 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.290994 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:38.291006 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:38.291333 (+ 327us) client_negotiation.cc:769] Sending connection context
0808 15:54:38.291652 (+ 319us) client_negotiation.cc:240] Negotiation successful
0808 15:54:38.292275 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":66,"thread_start_us":377,"threads_started":1}
I20260808 15:54:38.293265 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.270536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.27.84.254:42313)
0808 15:54:38.272511 (+ 1975us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:38.272517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:38.272562 (+ 45us) server_negotiation.cc:407] Connection header received
0808 15:54:38.274826 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:38.274831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:38.275073 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:38.275237 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:38.276902 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.278911 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.280426 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.281412 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.281631 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.285726 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:38.285734 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:38.285746 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:38.285775 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:38.288638 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.289609 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.289616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.289618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.289761 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.290235 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.290238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.290240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.290824 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:38.291001 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:38.291756 (+ 755us) server_negotiation.cc:299] Negotiation successful
0808 15:54:38.292052 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1783,"spinlock_wait_cycles":17792,"thread_start_us":1494,"threads_started":1}
I20260808 15:54:38.296968 31060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:38.298192 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.297674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56520)
0808 15:54:38.297888 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.298021 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260808 15:54:38.298527 30982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.27.84.253@KRBTEST.COM; defaulting to no policy
I20260808 15:54:38.300297 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.299822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56522)
0808 15:54:38.300004 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.300084 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260808 15:54:38.300637 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:36825: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab.
Entry for principal kudu/127.27.84.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab.
I20260808 15:54:38.344132 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:36825
--webserver_interface=127.27.84.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab
--principal=kudu/127.27.84.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42313,127.27.84.253:36825,127.27.84.252:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:54:38.350045 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } attempt: 1
I20260808 15:54:38.351366 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.350897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56532)
0808 15:54:38.351092 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.351232 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260808 15:54:38.352432 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.352038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56544)
0808 15:54:38.352195 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.352265 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260808 15:54:38.352625 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:36825: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
I20260808 15:54:38.420933 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } attempt: 2
I20260808 15:54:38.422281 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.421720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56548)
0808 15:54:38.421972 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.422093 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260808 15:54:38.423436 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.422989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56560)
0808 15:54:38.423155 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.423260 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260808 15:54:38.423666 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:36825: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
I20260808 15:54:38.514947 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } attempt: 3
I20260808 15:54:38.516465 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.515820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56566)
0808 15:54:38.516103 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.516241 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260808 15:54:38.517686 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.517227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56576)
0808 15:54:38.517426 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.517517 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260808 15:54:38.517920 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:36825: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
I20260808 15:54:38.661203 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } attempt: 4
I20260808 15:54:38.662467 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.661920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56592)
0808 15:54:38.662230 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.662323 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260808 15:54:38.663565 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.663193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56602)
0808 15:54:38.663361 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.663444 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 15:54:38.663760 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.253:36825: Network error: Client connection negotiation failed: client connection to 127.27.84.253:36825: connect: Connection refused (error 111)
W20260808 15:54:38.702039 31068 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:54:38.702483 31068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:38.702656 31068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:38.702797 31068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:38.710073 31068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:38.710198 31068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:38.710314 31068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:38.710402 31068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:38.710494 31068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:38.720649 31068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42313,127.27.84.253:36825,127.27.84.252:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.253.keytab
--principal=kudu/127.27.84.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.253:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.84.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:38.722177 31068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:38.723642 31068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:38.736059 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:38.736202 31081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:38.736377 31080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:38.737289 31068 server_base.cc:1034] running on GCE node
I20260808 15:54:38.738176 31068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:38.739867 31068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:38.741173 31068 hybrid_clock.cc:648] HybridClock initialized: now 1786204478741095 us; error 45 us; skew 500 ppm
I20260808 15:54:38.741564 31068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:38 dist-test-slave-twwt krb5kdc[30939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:38.745184 31068 init.cc:377] Logged in from keytab as kudu/127.27.84.253@KRBTEST.COM (short username kudu)
I20260808 15:54:38.746873 31068 webserver.cc:460] Webserver started at http://127.27.84.253:41769/ using document root <none> and password file <none>
I20260808 15:54:38.747592 31068 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:38.747686 31068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:38.747983 31068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:38.751228 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data/instance:
uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed"
format_stamp: "Formatted at 2026-08-08 15:54:38 on dist-test-slave-twwt"
server_key: "44525915c8b07f4e249c88e50f76907c"
server_key_iv: "54bd054068e11c350d1e736c00ac9703"
server_key_version: "encryptionkey@0"
I20260808 15:54:38.752033 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal/instance:
uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed"
format_stamp: "Formatted at 2026-08-08 15:54:38 on dist-test-slave-twwt"
server_key: "44525915c8b07f4e249c88e50f76907c"
server_key_iv: "54bd054068e11c350d1e736c00ac9703"
server_key_version: "encryptionkey@0"
I20260808 15:54:38.757656 31068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260808 15:54:38.761567 31095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:38.763260 31068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260808 15:54:38.763451 31068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal
uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed"
format_stamp: "Formatted at 2026-08-08 15:54:38 on dist-test-slave-twwt"
server_key: "44525915c8b07f4e249c88e50f76907c"
server_key_iv: "54bd054068e11c350d1e736c00ac9703"
server_key_version: "encryptionkey@0"
I20260808 15:54:38.763680 31068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:38.786222 31068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:38.790946 31068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:38.791254 31068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:38.810181 31068 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.253:36825
I20260808 15:54:38.810206 31168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.253:36825 every 8 connection(s)
I20260808 15:54:38.811662 31068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/data/info.pb
I20260808 15:54:38.815462 31169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260808 15:54:38.822011 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 31068
I20260808 15:54:38.822222 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-1/wal/instance
I20260808 15:54:38.822815 27987 external_mini_cluster.cc:1428] Setting key 6e78733fe29a55640eb6a2cf255cba56
I20260808 15:54:38.822229 31169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:54:38 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:38.845901 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.825545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41494 (local address 127.27.84.254:42313)
0808 15:54:38.826118 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:38.826126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:38.827230 (+ 1104us) server_negotiation.cc:407] Connection header received
0808 15:54:38.829200 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:38.829208 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:38.829308 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:38.829453 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:38.831109 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.832210 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.833731 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.834470 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.834639 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.838534 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:38.838540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:38.838543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:38.838590 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:38.842143 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.843074 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.843082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.843088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.843261 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.843711 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.843719 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.843725 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.844139 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:38.844309 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:38.845123 (+ 814us) server_negotiation.cc:299] Negotiation successful
0808 15:54:38.845399 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":261,"threads_started":1}
I20260808 15:54:38.846521 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.826160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42313 (local address 127.0.0.1:41494)
0808 15:54:38.827037 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.827084 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:38.828901 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:38.829655 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:38.829663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:38.830221 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:38.830882 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.830904 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.832376 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.832380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.833576 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.833597 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.834732 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.834738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.834948 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.835655 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:38.835677 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:38.838239 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:38.842321 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.842329 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.842343 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.842931 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.843394 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.843398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.843400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.843579 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.844283 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:38.844290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:38.844652 (+ 362us) client_negotiation.cc:769] Sending connection context
0808 15:54:38.844950 (+ 298us) client_negotiation.cc:240] Negotiation successful
0808 15:54:38.845343 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":391,"threads_started":1}
I20260808 15:54:38.848413 31169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:54:38 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.84.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.84.252@KRBTEST.COM" created.
I20260808 15:54:38.865258 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.849275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56610)
0808 15:54:38.849455 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.849475 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:38.849569 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:38.850678 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:38.850685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:38.851226 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:38.851478 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.851495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.852953 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.852961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.854534 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.854553 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.855422 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.855435 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.855649 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.856264 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:38.856292 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:38.858953 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:38.862421 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.862425 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.862429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.862917 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.863441 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.863445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.863447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.863601 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.864416 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:38.864425 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:38.864605 (+ 180us) client_negotiation.cc:769] Sending connection context
0808 15:54:38.864732 (+ 127us) client_negotiation.cc:240] Negotiation successful
0808 15:54:38.864989 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260808 15:54:38.865770 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.849560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56610 (local address 127.27.84.253:36825)
0808 15:54:38.850235 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:38.850241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:38.850264 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:54:38.850305 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:38.850319 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:38.850491 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:38.850648 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:38.851665 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.852805 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.854715 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.855251 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.855352 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.859170 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:38.859183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:38.859197 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:38.859228 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:38.862264 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.863079 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.863087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.863089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.863235 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.863751 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.863755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.863757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.864246 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:38.864464 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:38.864964 (+ 500us) server_negotiation.cc:299] Negotiation successful
0808 15:54:38.865266 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":384,"threads_started":1}
I20260808 15:54:38.868414 31169 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:38.869565 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.869134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45136)
0808 15:54:38.869310 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.869426 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260808 15:54:38.869807 31099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:38.871384 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.871039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45150)
0808 15:54:38.871206 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.871282 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260808 15:54:38.871577 31169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.84.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab.
Entry for principal kudu/127.27.84.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab.
I20260808 15:54:38.906842 27987 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
/tmp/dist-test-taskzIk6P5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:36177
--webserver_interface=127.27.84.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab
--principal=kudu/127.27.84.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.84.212:38355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.84.254:42313,127.27.84.253:36825,127.27.84.252:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260808 15:54:38.920984 31169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 1
I20260808 15:54:38.922235 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.921758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45164)
0808 15:54:38.921964 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.922079 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260808 15:54:38.923354 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.923004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45174)
0808 15:54:38.923165 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.923241 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:54:38.923642 31169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:38.962998 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } attempt: 5
Aug 08 15:54:38 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:38.977029 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.963927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.27.84.253:36825)
0808 15:54:38.964115 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:38.964123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:38.964152 (+ 29us) server_negotiation.cc:407] Connection header received
0808 15:54:38.964402 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:38.964407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:38.964491 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:38.964602 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:38.965476 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.966190 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.967480 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.967948 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.968087 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.971389 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:38.971394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:38.971397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:38.971430 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:38.974170 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.974952 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.974958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.974964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.975100 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:38.975517 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:38.975521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:38.975523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:38.975832 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:38.975998 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:38.976439 (+ 441us) server_negotiation.cc:299] Negotiation successful
0808 15:54:38.976670 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260808 15:54:38.977029 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.963828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56626)
0808 15:54:38.964065 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.964088 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:38.964241 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:38.964645 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:38.964650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:38.965096 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:38.965289 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.965299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.966336 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.966341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.967328 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:38.967341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:38.968114 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:38.968119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:38.968379 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:38.969048 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:38.969067 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:38.971164 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:38.974324 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.974330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.974335 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.974828 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.975229 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:38.975234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:38.975237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:38.975384 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:38.975961 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:38.975966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:38.976135 (+ 169us) client_negotiation.cc:769] Sending connection context
0808 15:54:38.976311 (+ 176us) client_negotiation.cc:240] Negotiation successful
0808 15:54:38.976634 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260808 15:54:38.978204 31060 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:38.979528 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.979117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45182)
0808 15:54:38.979304 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.979392 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260808 15:54:38.980615 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.980234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45194)
0808 15:54:38.980415 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.980517 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260808 15:54:38.980892 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:38.991884 31169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 2
I20260808 15:54:38.993100 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.992632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45198)
0808 15:54:38.992847 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.992939 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260808 15:54:38.994136 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:38.993782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45204)
0808 15:54:38.993948 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:38.994017 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260808 15:54:38.994357 31169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:39.032156 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 1
I20260808 15:54:39.033429 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.032950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45216)
0808 15:54:39.033157 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.033253 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260808 15:54:39.034457 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.034124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45218)
0808 15:54:39.034277 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.034351 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:54:39.034675 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:39.085575 31169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 3
I20260808 15:54:39.086869 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.086304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45234)
0808 15:54:39.086538 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.086685 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 15:54:39.087965 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.087522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45236)
0808 15:54:39.087741 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.087847 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260808 15:54:39.088171 31169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:39.102886 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 2
I20260808 15:54:39.104152 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.103600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45238)
0808 15:54:39.103881 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.103974 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260808 15:54:39.105160 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.104814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45250)
0808 15:54:39.104969 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.105048 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260808 15:54:39.105372 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:39.211585 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 3
I20260808 15:54:39.212762 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.212260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45254)
0808 15:54:39.212473 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.212573 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260808 15:54:39.213795 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.213425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45260)
0808 15:54:39.213625 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.213690 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260808 15:54:39.213990 31060 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
I20260808 15:54:39.231398 31169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 4
I20260808 15:54:39.232583 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.232067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45268)
0808 15:54:39.232331 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.232417 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260808 15:54:39.233572 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.233245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45274)
0808 15:54:39.233389 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.233466 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260808 15:54:39.233783 31169 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.84.252:36177: Network error: Client connection negotiation failed: client connection to 127.27.84.252:36177: connect: Connection refused (error 111)
W20260808 15:54:39.276963 31177 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260808 15:54:39.277372 31177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260808 15:54:39.277504 31177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260808 15:54:39.277644 31177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260808 15:54:39.285807 31177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260808 15:54:39.285943 31177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260808 15:54:39.286048 31177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260808 15:54:39.286145 31177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260808 15:54:39.286238 31177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260808 15:54:39.296339 31177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.84.212:38355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.84.254:42313,127.27.84.253:36825,127.27.84.252:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/krb5kdc/kudu_127.27.84.252.keytab
--principal=kudu/127.27.84.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.84.252:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.84.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 08 Aug 2026 14:12:59 UTC on ccycloud.quasar-aznhqa.root.comops.site
build id 4574924
ASAN enabled
I20260808 15:54:39.297791 31177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260808 15:54:39.299247 31177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260808 15:54:39.312455 31185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:39.312455 31183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260808 15:54:39.312485 31191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260808 15:54:39.313513 31177 server_base.cc:1034] running on GCE node
I20260808 15:54:39.314389 31177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260808 15:54:39.315965 31177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260808 15:54:39.317220 31177 hybrid_clock.cc:648] HybridClock initialized: now 1786204479317151 us; error 43 us; skew 500 ppm
I20260808 15:54:39.317574 31177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260808 15:54:39.320988 31177 init.cc:377] Logged in from keytab as kudu/127.27.84.252@KRBTEST.COM (short username kudu)
I20260808 15:54:39.322736 31177 webserver.cc:460] Webserver started at http://127.27.84.252:44197/ using document root <none> and password file <none>
I20260808 15:54:39.323473 31177 fs_manager.cc:359] Metadata directory not provided
I20260808 15:54:39.323594 31177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260808 15:54:39.323954 31177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260808 15:54:39.327203 31177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data/instance:
uuid: "7a62f580133747c6b7d80a79f9cc372c"
format_stamp: "Formatted at 2026-08-08 15:54:39 on dist-test-slave-twwt"
server_key: "7702711049900b85f7f0d47b62c2596f"
server_key_iv: "0e3f0ed30040615bfd67cc18b0a86309"
server_key_version: "encryptionkey@0"
I20260808 15:54:39.328083 31177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal/instance:
uuid: "7a62f580133747c6b7d80a79f9cc372c"
format_stamp: "Formatted at 2026-08-08 15:54:39 on dist-test-slave-twwt"
server_key: "7702711049900b85f7f0d47b62c2596f"
server_key_iv: "0e3f0ed30040615bfd67cc18b0a86309"
server_key_version: "encryptionkey@0"
I20260808 15:54:39.333487 31177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260808 15:54:39.337025 31207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260808 15:54:39.338382 31177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260808 15:54:39.338570 31177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data,/tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal
uuid: "7a62f580133747c6b7d80a79f9cc372c"
format_stamp: "Formatted at 2026-08-08 15:54:39 on dist-test-slave-twwt"
server_key: "7702711049900b85f7f0d47b62c2596f"
server_key_iv: "0e3f0ed30040615bfd67cc18b0a86309"
server_key_version: "encryptionkey@0"
I20260808 15:54:39.338862 31177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260808 15:54:39.364539 31177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260808 15:54:39.369055 31177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260808 15:54:39.369369 31177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260808 15:54:39.389114 31177 rpc_server.cc:307] RPC server started. Bound to: 127.27.84.252:36177
I20260808 15:54:39.389138 31287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.84.252:36177 every 8 connection(s)
I20260808 15:54:39.390725 31177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/data/info.pb
I20260808 15:54:39.391182 31060 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 4
I20260808 15:54:39.394033 27987 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu as pid 31177
I20260808 15:54:39.394224 27987 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskzIk6P5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786204089698720-27987-0/minicluster-data/master-2/wal/instance
I20260808 15:54:39.394799 27987 external_mini_cluster.cc:1428] Setting key 5d285b3a63ba21afdddafe5148e87345
I20260808 15:54:39.395644 31289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.254@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:54:39.409834 31289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:39.420367 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.391954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45290)
0808 15:54:39.392161 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.392182 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:39.392301 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:39.397722 (+ 5421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:39.397731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:39.398476 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:39.398810 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.398828 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.402810 (+ 3982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.402820 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.404747 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.404768 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.406798 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.406804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.406981 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.407665 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:39.407683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:39.410290 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:39.415815 (+ 5525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.415824 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.415832 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.416579 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.417310 (+ 731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.417318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.417324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.417539 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.418822 (+ 1283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:39.418831 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:39.419126 (+ 295us) client_negotiation.cc:769] Sending connection context
0808 15:54:39.419313 (+ 187us) client_negotiation.cc:240] Negotiation successful
0808 15:54:39.419747 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:54:39.424029 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.393102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.27.84.252:36177)
0808 15:54:39.393959 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.393971 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.394014 (+ 43us) server_negotiation.cc:407] Connection header received
0808 15:54:39.396435 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.396478 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.397149 (+ 671us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.399157 (+ 2008us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.399865 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.402533 (+ 2668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.405027 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.406533 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.407402 (+ 869us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.410651 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.410663 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.410684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.410739 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.415608 (+ 4869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.416809 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.416822 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.416835 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.417130 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.417715 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.417723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.417729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.418628 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.420816 (+ 2188us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.421340 (+ 524us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.422055 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":412,"threads_started":1}
I20260808 15:54:39.432802 31060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: Bootstrap starting.
I20260808 15:54:39.433368 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.395694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45298 (local address 127.27.84.252:36177)
0808 15:54:39.409493 (+ 13799us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.409501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.409528 (+ 27us) server_negotiation.cc:407] Connection header received
0808 15:54:39.409589 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.409594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.409670 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.409777 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.411962 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.413221 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.415888 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.416864 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.417031 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.425197 (+ 8166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.425207 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.425216 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.425277 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.429790 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.430803 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.430809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.430812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.431135 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.431669 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.431675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.431679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.432055 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.432392 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.432714 (+ 322us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.432951 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13630,"spinlock_wait_cycles":28544,"thread_start_us":13405,"threads_started":1}
I20260808 15:54:39.437083 31060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:39.438685 31060 log.cc:824] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: Log is configured to *not* fsync() on all Append() calls
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:39.442037 31060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: No bootstrap required, opened a new log
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.254@KRBTEST.COM
I20260808 15:54:39.447428 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.413198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.254:42313 (local address 127.0.0.1:41506)
0808 15:54:39.430204 (+ 17006us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.430242 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:39.430453 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:39.430889 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:39.430899 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:39.431977 (+ 1078us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:39.432317 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.432331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.433974 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.433977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.435389 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.435428 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.436811 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.436821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.437044 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.437772 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:39.437796 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:39.440542 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:39.444678 (+ 4136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.444689 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.444697 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.445076 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.445491 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.445496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.445500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.445647 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.446231 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:39.446239 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:39.446418 (+ 179us) client_negotiation.cc:769] Sending connection context
0808 15:54:39.446596 (+ 178us) client_negotiation.cc:240] Negotiation successful
0808 15:54:39.446978 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16741,"thread_start_us":15352,"threads_started":1}
I20260808 15:54:39.447430 31295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.413232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.27.84.254:42313)
0808 15:54:39.427168 (+ 13936us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.427177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.430381 (+ 3204us) server_negotiation.cc:407] Connection header received
0808 15:54:39.430587 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.430592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.430712 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.430819 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.432493 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.433760 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.435615 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.436582 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.436756 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.440772 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.440779 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.440783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.440821 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.444529 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.445212 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.445217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.445221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.445353 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.445778 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.445783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.445785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.446075 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.446260 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.446750 (+ 490us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.446972 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13759,"thread_start_us":13625,"threads_started":1}
I20260808 15:54:39.450872 31289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } has no permanent_uuid. Determining permanent_uuid...
I20260808 15:54:39.454182 31060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.454803 31060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 329566f2f6e847b1ba03b80f676f31cf, State: Initialized, Role: FOLLOWER
I20260808 15:54:39.455603 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.435554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41520 (local address 127.27.84.254:42313)
0808 15:54:39.437426 (+ 1872us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.437433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.437456 (+ 23us) server_negotiation.cc:407] Connection header received
0808 15:54:39.437509 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.437515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.437621 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.437772 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.439107 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.440377 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.442309 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.443169 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.443382 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.449048 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.449053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.449056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.449088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.452784 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.453469 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.453475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.453480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.453617 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.454016 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.454022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.454025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.454304 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.454639 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.454942 (+ 303us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.455165 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1723,"thread_start_us":1555,"threads_started":1}
I20260808 15:54:39.455443 31060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:39.462455 31300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.462963 31300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:39.464888 31060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: configured and running, proceeding with master startup.
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.84.253@KRBTEST.COM
I20260808 15:54:39.469393 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.451675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.253:36825 (local address 127.0.0.1:56632)
0808 15:54:39.451952 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.451972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:39.452083 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:39.452482 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:39.452487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:39.452974 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:39.453157 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.453169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.454588 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.454593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.455961 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.455984 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.457038 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.457047 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.457335 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.458134 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:39.458166 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:39.461534 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:39.465933 (+ 4399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.465942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.465947 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.466629 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.467243 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.467251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.467257 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.467510 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.468267 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:39.468271 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:39.468503 (+ 232us) client_negotiation.cc:769] Sending connection context
0808 15:54:39.468668 (+ 165us) client_negotiation.cc:240] Negotiation successful
0808 15:54:39.469007 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260808 15:54:39.469429 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.451787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.27.84.253:36825)
0808 15:54:39.452011 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.452017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.452037 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:54:39.452228 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.452234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.452321 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.452433 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.453335 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.454348 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.456218 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.456830 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.456973 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.461825 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.461834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.461842 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.461897 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.465763 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.466810 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.466816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.466819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.467041 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.467677 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.467685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.467690 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.468047 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.468206 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.468795 (+ 589us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.469013 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260808 15:54:39.470780 31289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } has no permanent_uuid. Determining permanent_uuid...
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.252@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:54:39.480212 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.457450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.27.84.253:36825)
0808 15:54:39.459077 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.459081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.459100 (+ 19us) server_negotiation.cc:407] Connection header received
0808 15:54:39.459138 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.459141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.459192 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.459309 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.460888 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.461649 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.462992 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.463667 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.463811 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.471034 (+ 7223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.471042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.471046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.471084 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.475245 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.476415 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.476438 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.476444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.476654 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.477506 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.477511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.477514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.477890 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.478165 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.479504 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.479748 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1536,"thread_start_us":1333,"threads_started":1}
I20260808 15:54:39.485704 27987 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260808 15:54:39.487246 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.472131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.27.84.252:36177)
0808 15:54:39.472310 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.472315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.472332 (+ 17us) server_negotiation.cc:407] Connection header received
0808 15:54:39.472418 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.472423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.472493 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.472593 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.473369 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.474507 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.475972 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.476404 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.476529 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.480811 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.480817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.480822 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.480885 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.484296 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.485091 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.485096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.485099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.485228 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.485622 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.485626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.485630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.486016 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.486146 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.486674 (+ 528us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.486916 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260808 15:54:39.487387 30987 tablet.cc:2369] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:54:39.488060 31297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.471715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45302)
0808 15:54:39.472188 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.472209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:39.472319 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:39.472609 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:39.472613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:39.473065 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:39.473214 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.473223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.474697 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.474704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.475829 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.475841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.476549 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.476554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.476761 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.477389 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:39.477417 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:39.480219 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:39.484436 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.484443 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.484447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.484968 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.485356 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.485360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.485363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.485509 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.486228 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:39.486235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:39.486375 (+ 140us) client_negotiation.cc:769] Sending connection context
0808 15:54:39.487419 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0808 15:54:39.487726 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348}
I20260808 15:54:39.488994 31024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "329566f2f6e847b1ba03b80f676f31cf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41520
I20260808 15:54:39.489409 31024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260808 15:54:39.489624 31024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:39.493482 31024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.494340 31289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: Bootstrap starting.
I20260808 15:54:39.483096 30961 master_runner.cc:186] Error getting master registration for 127.27.84.253:36825: OK, instance_id { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" instance_seqno: 1786204478804922 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:54:39.496466 30961 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:54:39.497264 31024 leader_election.cc:288] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [CANDIDATE]: Term 1 election: Requested vote from peers fc9a8186a6ca49a7a5fd892498a6e0ed (127.27.84.253:36825), 7a62f580133747c6b7d80a79f9cc372c (127.27.84.252:36177)
I20260808 15:54:39.498180 31289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:39.499598 31289 log.cc:824] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: Log is configured to *not* fsync() on all Append() calls
W20260808 15:54:39.504006 31317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:54:39.504426 31317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:54:39.518496 31289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: No bootstrap required, opened a new log
I20260808 15:54:39.530195 31289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.530695 31289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a62f580133747c6b7d80a79f9cc372c, State: Initialized, Role: FOLLOWER
I20260808 15:54:39.531257 31289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.532974 31169 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } attempt: 5
I20260808 15:54:39.533891 31289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:39.535145 31250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "329566f2f6e847b1ba03b80f676f31cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a62f580133747c6b7d80a79f9cc372c"
I20260808 15:54:39.535722 31250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:39.540391 31250 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329566f2f6e847b1ba03b80f676f31cf in term 1.
I20260808 15:54:39.541271 30983 leader_election.cc:302] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 329566f2f6e847b1ba03b80f676f31cf, 7a62f580133747c6b7d80a79f9cc372c; no voters:
I20260808 15:54:39.541855 31300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 1 FOLLOWER]: Leader election won for term 1
Aug 08 15:54:39 dist-test-slave-twwt krb5kdc[30939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786204478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.84.253@KRBTEST.COM for kudu/127.27.84.252@KRBTEST.COM
I20260808 15:54:39.532757 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.568171 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:39.576797 31292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.534459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.27.84.252:36177)
0808 15:54:39.534656 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.534662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.534682 (+ 20us) server_negotiation.cc:407] Connection header received
0808 15:54:39.534748 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.534753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.534828 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.534929 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.544536 (+ 9607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.545874 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.548570 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.549517 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.549668 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.559084 (+ 9416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.559094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.559102 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.559161 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.566035 (+ 6874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.569836 (+ 3801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.569874 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.569881 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.570283 (+ 402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.571343 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.571352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.571357 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.572290 (+ 933us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.572724 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.573923 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.575002 (+ 1079us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:54:39.577373 31300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [term 1 LEADER]: Becoming Leader. State: Replica: 329566f2f6e847b1ba03b80f676f31cf, State: Running, Role: LEADER
I20260808 15:54:39.578133 31300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.583979 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.533669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.84.252:36177 (local address 127.0.0.1:45304)
0808 15:54:39.534084 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0808 15:54:39.534106 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0808 15:54:39.534234 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0808 15:54:39.542307 (+ 8073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0808 15:54:39.542317 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0808 15:54:39.543062 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0808 15:54:39.543391 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.543408 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.546231 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.546241 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.548290 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0808 15:54:39.548313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.550052 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.550061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0808 15:54:39.550598 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.551788 (+ 1190us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0808 15:54:39.551848 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0808 15:54:39.558430 (+ 6582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0808 15:54:39.566722 (+ 8292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.566732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.566832 (+ 100us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.569600 (+ 2768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.570632 (+ 1032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0808 15:54:39.570641 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0808 15:54:39.570647 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0808 15:54:39.571060 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0808 15:54:39.572907 (+ 1847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0808 15:54:39.572915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0808 15:54:39.573265 (+ 350us) client_negotiation.cc:769] Sending connection context
0808 15:54:39.575414 (+ 2149us) client_negotiation.cc:240] Negotiation successful
0808 15:54:39.577854 (+ 2440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260808 15:54:39.588359 31298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.541219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.27.84.254:42313)
0808 15:54:39.541401 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.541407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.558359 (+ 16952us) server_negotiation.cc:407] Connection header received
0808 15:54:39.558920 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.558927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.559052 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.559137 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.562725 (+ 3588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.564058 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.568856 (+ 4798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.569578 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.569681 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.578130 (+ 8449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.578137 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.578142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.578187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.583147 (+ 4960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.585557 (+ 2410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.585565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.585571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.585729 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.586282 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.586289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.586295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.586766 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.587017 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.587728 (+ 711us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.587994 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260808 15:54:39.593365 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.544009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.27.84.253:36825)
0808 15:54:39.544259 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.544267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.562334 (+ 18067us) server_negotiation.cc:407] Connection header received
0808 15:54:39.566460 (+ 4126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.566466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.566550 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.566643 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.570656 (+ 4013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.571863 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.574058 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.574720 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.574813 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.581063 (+ 6250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.581069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.581073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.581107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.584665 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.590595 (+ 5930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.590600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.590639 (+ 39us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.590783 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.591249 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.591257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.591263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.591648 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.591929 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.592649 (+ 720us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.592913 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260808 15:54:39.594839 31169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: Bootstrap starting.
I20260808 15:54:39.600605 31169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: Neither blocks nor log segments found. Creating new log.
I20260808 15:54:39.602322 31169 log.cc:824] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: Log is configured to *not* fsync() on all Append() calls
I20260808 15:54:39.605581 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329566f2f6e847b1ba03b80f676f31cf. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.606002 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:39.607203 31341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260808 15:54:39.608294 31291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:39.553015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.27.84.252:36177)
0808 15:54:39.553251 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.553260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.566004 (+ 12744us) server_negotiation.cc:407] Connection header received
0808 15:54:39.570884 (+ 4880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.570890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.570984 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.571053 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.573786 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.574939 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.584531 (+ 9592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:39.585285 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:39.585394 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:39.591086 (+ 5692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0808 15:54:39.591094 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0808 15:54:39.591099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0808 15:54:39.591144 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0808 15:54:39.594765 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.595770 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.595775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.595779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.595934 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0808 15:54:39.603170 (+ 7236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0808 15:54:39.603175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0808 15:54:39.603179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0808 15:54:39.603713 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0808 15:54:39.603987 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:39.605357 (+ 1370us) server_negotiation.cc:299] Negotiation successful
0808 15:54:39.607903 (+ 2546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260808 15:54:39.610176 31169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: No bootstrap required, opened a new log
I20260808 15:54:39.611609 31341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260808 15:54:39.621575 31215 tablet.cc:2369] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260808 15:54:39.625931 31169 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.626506 31169 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9a8186a6ca49a7a5fd892498a6e0ed, State: Initialized, Role: FOLLOWER
I20260808 15:54:39.627272 31169 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } }
I20260808 15:54:39.634330 31250 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [term 1 FOLLOWER]: Refusing update from remote peer 329566f2f6e847b1ba03b80f676f31cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:39.635629 31300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:39.639598 31132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [term 0 FOLLOWER]: Advancing to term 1
I20260808 15:54:39.640910 31135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "329566f2f6e847b1ba03b80f676f31cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed"
I20260808 15:54:39.644866 31132 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [term 1 FOLLOWER]: Refusing update from remote peer 329566f2f6e847b1ba03b80f676f31cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260808 15:54:39.649488 31135 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329566f2f6e847b1ba03b80f676f31cf in term 1.
I20260808 15:54:39.651038 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.651512 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:39.652287 31169 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: configured and running, proceeding with master startup.
I20260808 15:54:39.652249 31300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260808 15:54:39.673697 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329566f2f6e847b1ba03b80f676f31cf. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.674062 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:39.675148 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 329566f2f6e847b1ba03b80f676f31cf. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.675511 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:39.677976 31342 mvcc.cc:205] Tried to move back new op lower bound from 7316293548568829952 to 7316293548358914048. Current Snapshot: MvccSnapshot[applied={T|T < 7316293548568829952}]
I20260808 15:54:39.679404 31300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.679833 31300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:39.680054 31350 mvcc.cc:205] Tried to move back new op lower bound from 7316293548568829952 to 7316293548358914048. Current Snapshot: MvccSnapshot[applied={T|T < 7316293548568829952}]
I20260808 15:54:39.695025 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.695389 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf [sys.catalog]: This master's current role is: LEADER
I20260808 15:54:39.698228 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.698477 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed [sys.catalog]: This master's current role is: FOLLOWER
I20260808 15:54:39.711251 31341 catalog_manager.cc:1347] Generated new cluster ID: d1b279776d51473d8477f8565e12e405
I20260808 15:54:39.711377 31341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260808 15:54:39.687817 31068 master_runner.cc:186] Error getting master registration for 127.27.84.252:36177: OK, instance_id { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" instance_seqno: 1786204479383050 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260808 15:54:39.715371 31068 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260808 15:54:39.716086 31324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "329566f2f6e847b1ba03b80f676f31cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "329566f2f6e847b1ba03b80f676f31cf" member_type: VOTER last_known_addr { host: "127.27.84.254" port: 42313 } } peers { permanent_uuid: "fc9a8186a6ca49a7a5fd892498a6e0ed" member_type: VOTER last_known_addr { host: "127.27.84.253" port: 36825 } } peers { permanent_uuid: "7a62f580133747c6b7d80a79f9cc372c" member_type: VOTER last_known_addr { host: "127.27.84.252" port: 36177 } } }
I20260808 15:54:39.716364 31324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c [sys.catalog]: This master's current role is: FOLLOWER
W20260808 15:54:39.738857 31385 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260808 15:54:39.739045 31385 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:54:39.739276 31389 catalog_manager.cc:1259] Loaded cluster ID: d1b279776d51473d8477f8565e12e405
I20260808 15:54:39.739420 31389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: loading cluster ID for follower catalog manager: success
W20260808 15:54:39.740716 31389 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260808 15:54:39.740866 31389 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260808 15:54:39.749341 31341 catalog_manager.cc:1370] Generated new certificate authority record
I20260808 15:54:39.751369 31341 catalog_manager.cc:1504] Loading token signing keys...
I20260808 15:54:39.782227 31341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 329566f2f6e847b1ba03b80f676f31cf: Generated new TSK 0
I20260808 15:54:39.783550 31341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260808 15:54:39.959110 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:39.937380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.27.84.253:36825)
0808 15:54:39.937873 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.937882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.948914 (+ 11032us) server_negotiation.cc:407] Connection header received
0808 15:54:39.955375 (+ 6461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.955387 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.955542 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.955633 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.958821 (+ 3188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56668: BlockingRecv error: recv got EOF from 127.0.0.1:56668 (error 108)
Metrics: {"server-negotiator.queue_time_us":329}
W20260808 15:54:39.963583 31298 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:39.937171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.27.84.254:42313)
0808 15:54:39.937669 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.937678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.961085 (+ 23407us) server_negotiation.cc:407] Connection header received
0808 15:54:39.961503 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.961512 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.961628 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.961719 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.963290 (+ 1571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41538: BlockingRecv error: recv got EOF from 127.0.0.1:41538 (error 108)
Metrics: {"server-negotiator.queue_time_us":295}
W20260808 15:54:39.965498 31291 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:39.944782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45326 (local address 127.27.84.252:36177)
0808 15:54:39.945069 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:39.945078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:39.962918 (+ 17840us) server_negotiation.cc:407] Connection header received
0808 15:54:39.963780 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:39.963786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:39.963882 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:39.963968 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:39.965228 (+ 1260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45326: BlockingRecv error: recv got EOF from 127.0.0.1:45326 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260808 15:54:40.029542 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.016674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.27.84.253:36825)
0808 15:54:40.016881 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.016888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.027361 (+ 10473us) server_negotiation.cc:407] Connection header received
0808 15:54:40.027654 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.027662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.027845 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.027986 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.029247 (+ 1261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56672: BlockingRecv error: recv got EOF from 127.0.0.1:56672 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260808 15:54:40.080979 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.068826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.27.84.253:36825)
0808 15:54:40.069042 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.069049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.078819 (+ 9770us) server_negotiation.cc:407] Connection header received
0808 15:54:40.079392 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.079399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.079497 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.079574 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.080682 (+ 1108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56674: BlockingRecv error: recv got EOF from 127.0.0.1:56674 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260808 15:54:40.135371 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.121601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.27.84.253:36825)
0808 15:54:40.121807 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.121815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.133153 (+ 11338us) server_negotiation.cc:407] Connection header received
0808 15:54:40.133527 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.133534 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.133637 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.133742 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.135075 (+ 1333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56680: BlockingRecv error: recv got EOF from 127.0.0.1:56680 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:54:40.188378 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.176091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.27.84.253:36825)
0808 15:54:40.176318 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.176325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.186421 (+ 10096us) server_negotiation.cc:407] Connection header received
0808 15:54:40.186803 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.186812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.186915 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.186994 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.188107 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56686: BlockingRecv error: recv got EOF from 127.0.0.1:56686 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260808 15:54:40.252442 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.239526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.27.84.253:36825)
0808 15:54:40.239731 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.239738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.250370 (+ 10632us) server_negotiation.cc:407] Connection header received
0808 15:54:40.250703 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.250710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.250855 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.250949 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.252154 (+ 1205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56694: BlockingRecv error: recv got EOF from 127.0.0.1:56694 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260808 15:54:40.279985 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.272791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.27.84.253:36825)
0808 15:54:40.273058 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.273068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.278330 (+ 5262us) server_negotiation.cc:407] Connection header received
0808 15:54:40.278584 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.278589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.278729 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.278853 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.279699 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56700: BlockingRecv error: recv got EOF from 127.0.0.1:56700 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260808 15:54:40.319177 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.316715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.27.84.253:36825)
0808 15:54:40.316986 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.316999 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.317595 (+ 596us) server_negotiation.cc:407] Connection header received
0808 15:54:40.317843 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.317848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.317931 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.318054 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.318892 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56708: BlockingRecv error: recv got EOF from 127.0.0.1:56708 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260808 15:54:40.389084 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.387037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.27.84.253:36825)
0808 15:54:40.387231 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.387235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.387693 (+ 458us) server_negotiation.cc:407] Connection header received
0808 15:54:40.387874 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.387878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.387953 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.388051 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.388846 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56722: BlockingRecv error: recv got EOF from 127.0.0.1:56722 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260808 15:54:40.527556 31304 negotiation.cc:337] Failed RPC negotiation. Trace:
0808 15:54:40.525625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.27.84.253:36825)
0808 15:54:40.525811 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.525815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.526165 (+ 350us) server_negotiation.cc:407] Connection header received
0808 15:54:40.526346 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.526350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.526433 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.526527 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0808 15:54:40.527294 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56724: BlockingRecv error: recv got EOF from 127.0.0.1:56724 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260808 15:54:40.740852 31385 catalog_manager.cc:1259] Loaded cluster ID: d1b279776d51473d8477f8565e12e405
I20260808 15:54:40.740969 31385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: loading cluster ID for follower catalog manager: success
I20260808 15:54:40.743849 31385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: acquiring CA information for follower catalog manager: success
I20260808 15:54:40.744196 31389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: acquiring CA information for follower catalog manager: success
I20260808 15:54:40.745841 31385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc9a8186a6ca49a7a5fd892498a6e0ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:54:40.745903 31389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a62f580133747c6b7d80a79f9cc372c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260808 15:54:40.793447 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:40.787738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.27.84.253:36825)
0808 15:54:40.787995 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.788002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.788163 (+ 161us) server_negotiation.cc:407] Connection header received
0808 15:54:40.788336 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.788341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.788514 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.788620 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:40.789038 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.789778 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.791199 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.791593 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.791690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:40.792194 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:40.792794 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:40.792909 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:40.793020 (+ 111us) server_negotiation.cc:299] Negotiation successful
0808 15:54:40.793098 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260808 15:54:40.808339 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:40.799929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.27.84.252:36177)
0808 15:54:40.802991 (+ 3062us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.802996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.803014 (+ 18us) server_negotiation.cc:407] Connection header received
0808 15:54:40.803114 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.803118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.803271 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.803428 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:40.803830 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.804589 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.806031 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.806453 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.806546 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:40.807130 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:40.807680 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:40.807800 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:40.807901 (+ 101us) server_negotiation.cc:299] Negotiation successful
0808 15:54:40.807974 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2949,"thread_start_us":2841,"threads_started":1}
I20260808 15:54:40.833413 31304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:40.819862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.27.84.253:36825)
0808 15:54:40.820063 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.820069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.826889 (+ 6820us) server_negotiation.cc:407] Connection header received
0808 15:54:40.827168 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.827174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.827268 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.827358 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:40.828072 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.829166 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.831516 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.831952 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.832103 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:40.832634 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:40.832760 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:40.832868 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:40.833028 (+ 160us) server_negotiation.cc:299] Negotiation successful
0808 15:54:40.833116 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260808 15:54:40.836463 31491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:40.819643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.27.84.252:36177)
0808 15:54:40.819903 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.819912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.829544 (+ 9632us) server_negotiation.cc:407] Connection header received
0808 15:54:40.829766 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.829776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.829887 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.829978 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:40.830819 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.831969 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.834334 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.835055 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.835169 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:40.835695 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:40.835806 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:40.835920 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:40.836069 (+ 149us) server_negotiation.cc:299] Negotiation successful
0808 15:54:40.836151 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260808 15:54:40.837819 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0808 15:54:40.819579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.27.84.254:42313)
0808 15:54:40.828432 (+ 8853us) server_negotiation.cc:206] Beginning negotiation
0808 15:54:40.828439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0808 15:54:40.829032 (+ 593us) server_negotiation.cc:407] Connection header received
0808 15:54:40.829924 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0808 15:54:40.829933 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0808 15:54:40.830179 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0808 15:54:40.830421 (+ 242us) server_negotiation.cc:226] Negotiated authn=TOKEN
0808 15:54:40.831335 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.832407 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.835320 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0808 15:54:40.835959 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0808 15:54:40.836078 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0808 15:54:40.836878 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0808 15:54:40.837213 (+ 335us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0808 15:54:40.837358 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0808 15:54:40.837468 (+ 110us) server_negotiation.cc:299] Negotiation successful
0808 15:54:40.837553 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8707,"thread_start_us":9681,"threads_started":1}
I20260808 15:54:40.842128 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 30961
I20260808 15:54:40.843374 31054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:40.956635 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 30961
W20260808 15:54:40.988621 31100 connection.cc:535] server connection from 127.0.0.1:56626 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260808 15:54:40.988672 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31068
I20260808 15:54:40.989893 31164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:41.101881 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 31068
I20260808 15:54:41.130710 27987 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu pid 31177
I20260808 15:54:41.131913 31279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260808 15:54:41.240605 27987 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskzIk6P5/build/asan/bin/kudu with pid 31177
2026-08-08T15:54:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5288 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10716 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (391513 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST